Skip to content

Commit 36679a3

Browse files
martatolosamartatolosa
authored
Merge pull request #19 from oracle-devrel/timeseris
reformat names
2 parents 7f1dda3 + c61589e commit 36679a3

5 files changed

Lines changed: 163 additions & 21 deletions

File tree

‎IoTDBJDBC/.classpath‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -26,7 +26,7 @@
2626
<attribute name="optional" value="true"/>
2727
</attributes>
2828
</classpathentry>
29-
<classpathentry kind="con" path="org.eclipse.jdt.launching.JRE_CONTAINER/org.eclipse.jdt.internal.debug.ui.launcher.StandardVMType/JavaSE-17">
29+
<classpathentry kind="con" path="org.eclipse.jdt.launching.JRE_CONTAINER/org.eclipse.jdt.internal.debug.ui.launcher.StandardVMType/JavaSE-21">
3030
<attributes>
3131
<attribute name="maven.pomderived" value="true"/>
3232
</attributes>

‎IoTDBJDBC/SetupNotes/PromQL.txt‎

Lines changed: 79 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,79 @@
1+
Login to the telementry database as a user with query permisions
2+
To get the exposed packages execute
3+
4+
desc DBMS_ADB_TELEMETRY_QUERY
5+
6+
Note that all of the following REQUIRE absolute timestamps Unix Epoch offsets to get that look at https://www.epochconverter.com/ (or risk chatGPT)
7+
8+
this should get the most recent value before the provided timestamp (within 5 mins) for the current charge percentage
9+
Note The OTEL Naming documents say the use of . as a grouping separator is OK (as long as there are not two concurrent dots - so no ..)
10+
it does however recomend against the use of upper case (it says should, not must though) and recommends the use of snake case - so the
11+
metric name would be iot.normalized.current_charge_percentage and the code automaticallu converts any names (attribute key, metric
12+
name, attribute name, event name) into dot separated snake_case so
13+
IN theory a promql would be something like this
14+
select dbms_ADB_telemetry_query.PROMQL_INSTANT('iot.normalized.current_charge_percentage', 1782926574) from dual
15+
HOWEVER
16+
currently there seems to be a bug in the promql parsing code that can't handle the simple from using dot
17+
separation, so for now we need to do something like
18+
select dbms_cloud_telemetry_query.promql_instant('{__name__="iot.normalized.current_charge_percentage"}',1782928327) from dual;
19+
20+
ADDITIONALLY
21+
is restricting based on tags there is an additional problem (probabaly a bug to be honest) in that the names (key) of the attributes
22+
have any . in the name replaced with _, so iot.digital_twin.model_name becoms a tag of iot_digital_twin_model_name which rather messes
23+
up the naming hierarchy. Engineering are currently looking to see if this is a ingest "undocumented feature" or a deliberate choice, but
24+
it's clear from the OTLP naming rules ( https://opentelemetry.io/docs/specs/semconv/general/naming/ ) that . is allowed as a namespace separator
25+
26+
Thge PROMSQL procedures and their arguments are below, note that you can name a param using name -> value, or use positional params
27+
based on the order below, all CLOB's are JSOB formatted prometheus structures.
28+
The promql is defined at https://prometheus.io/docs/prometheus/latest/querying/basics/
29+
30+
31+
FUNCTION PROMQL_INSTANT RETURNS CLOB
32+
the last value of the metric BEFORE or at the time stamp (with a max of 300 seconds prior)
33+
34+
Argument Name Type In/Out Default?
35+
------------------------- -------------------- ------ --------
36+
PROMQL_QUERY VARCHAR2 IN
37+
INSTANT_TIME_EPOCH NUMBER IN
38+
FETCH_SQL NUMBER IN Y
39+
40+
FUNCTION PROMQL_LABEL RETURNS CLOB
41+
Get the labels (need more info on this)
42+
43+
Argument Name Type In/Out Default?
44+
------------------------- -------------------- ------ --------
45+
PROMQL_QUERY VARCHAR2 IN
46+
START_TIME_EPOCH NUMBER IN
47+
END_TIME_EPOCH NUMBER IN
48+
FETCH_SQL NUMBER IN Y
49+
50+
FUNCTION PROMQL_RANGE RETURNS CLOB
51+
The values in the given range
52+
STEP_SIZE_SECONDS sets the size of the step, which returns a value for each step (like
53+
the instant) between the start and end time (defaults to 10 seconds). This is basically
54+
the PROMQL_INSTANT run once for each step.
55+
56+
Argument Name Type In/Out Default?
57+
------------------------- -------------------- ------ --------
58+
PROMQL_QUERY VARCHAR2 IN
59+
START_TIME_EPOCH NUMBER IN
60+
END_TIME_EPOCH NUMBER IN
61+
STEP_SIZE_SECONDS NUMBER IN
62+
FETCH_SQL NUMBER IN Y
63+
64+
FUNCTION PROMQL_SERIES RETURNS CLOB
65+
Get the labels (need more info on this)
66+
Argument Name Type In/Out Default?
67+
------------------------- -------------------- ------ --------
68+
PROMQL_QUERY VARCHAR2 IN
69+
START_TIME_EPOCH NUMBER IN
70+
END_TIME_EPOCH NUMBER IN
71+
FETCH_SQL NUMBER IN Y
72+
73+
example (using the current limitations on naming)
74+
select dbms_cloud_telemetry_query.promql_instant('{__name__="iot.normalized.inverter_power_watts_point_in_time"}',1782928327) from dual;
75+
76+
getting a set of ranges using the default step of 10
77+
select dbms_cloud_telemetry_query.promql_range('{__name__="iot.normalized.InverterPowerWattsPointInTime"}',1782925327, 1782928327) from dual;
78+
79+
getting a set of ranged using a step of 100

‎IoTDBJDBC/pom.xml‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -213,6 +213,11 @@ SOFTWARE. -->
213213
<artifactId>jakarta.validation-api</artifactId>
214214
<scope>compile</scope>
215215
</dependency>
216+
<dependency>
217+
<groupId>com.google.guava</groupId>
218+
<artifactId>guava</artifactId>
219+
<scope>compile</scope>
220+
</dependency>
216221

217222
</dependencies>
218223
<build>

‎IoTDBJDBC/src/main/java/com/oracle/demo/timg/iot/iotdbjdbc/messagehandler/outputs/http/normalizeddata/timeseriesdb/TimeSeriesDBOutputOTLP.java‎

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -137,8 +137,7 @@ public NormalizedData[] processNormalizedData(NormalizedData normalizedData) thr
137137
}
138138

139139
private void addScopeAttributes(NormalizedDataMetricsDataBuilder builder) {
140-
// for now this does nothing
141-
// builder.scopeAttribute("my.scope.attribute", "some scope attribute");
140+
// for now none to add
142141
}
143142

144143
private void addResourceAttributes(NormalizedDataMetricsDataBuilder builder, NormalizedData normalizedData) {
@@ -147,10 +146,12 @@ private void addResourceAttributes(NormalizedDataMetricsDataBuilder builder, Nor
147146
try {
148147
String modelId = deviceModelInstancesCache.getModelIdByInstanceId(instanceId, true);
149148
builder.resourceAttribute("iot.digital_twin.model_id", modelId);
149+
log.info("Added model ID resource attribute of " + modelId);
150150
String modelName;
151151
try {
152152
modelName = deviceModelInstancesCache.getModelNameByModelId(modelId, true);
153153
builder.resourceAttribute("iot.digital_twin.model_name", modelName);
154+
log.info("Added model name resource attribute of " + modelName);
154155
} catch (MissingModelException e) {
155156
log.severe("No model name found for modelid " + modelId);
156157
} catch (SQLException e) {

‎IoTDBJDBC/src/main/java/com/oracle/demo/timg/iot/iotdbjdbc/messagehandler/outputs/http/normalizeddata/timeseriesdb/otlp/NormalizedDataMetricsDataBuilder.java‎

Lines changed: 75 additions & 18 deletions
Original file line numberDiff line numberDiff line change
@@ -98,42 +98,42 @@ public NormalizedDataMetricsDataBuilder metric(String metricPrefix, String metri
9898

9999
public NormalizedDataMetricsDataBuilder resourceAttribute(String key, String value) {
100100
if (hasText(key)) {
101-
resourceAttributes.add(OtlpAttributeUtils.attribute(key, value));
101+
resourceAttributes.add(attribute(key, value));
102102
}
103103
return this;
104104
}
105105

106106
public NormalizedDataMetricsDataBuilder resourceAttribute(String key, AnyValue value) {
107107
if (hasText(key) && value != null) {
108-
resourceAttributes.add(OtlpAttributeUtils.attribute(key, value));
108+
resourceAttributes.add(attribute(key, value));
109109
}
110110
return this;
111111
}
112112

113113
public NormalizedDataMetricsDataBuilder resourceAttributes(List<KeyValue> attributes) {
114114
if (attributes != null) {
115-
resourceAttributes.addAll(attributes);
115+
resourceAttributes.addAll(normalizeAttributeKeys(attributes));
116116
}
117117
return this;
118118
}
119119

120120
public NormalizedDataMetricsDataBuilder scopeAttribute(String key, String value) {
121121
if (hasText(key)) {
122-
scopeAttributes.add(OtlpAttributeUtils.attribute(key, value));
122+
scopeAttributes.add(attribute(key, value));
123123
}
124124
return this;
125125
}
126126

127127
public NormalizedDataMetricsDataBuilder scopeAttribute(String key, AnyValue value) {
128128
if (hasText(key) && value != null) {
129-
scopeAttributes.add(OtlpAttributeUtils.attribute(key, value));
129+
scopeAttributes.add(attribute(key, value));
130130
}
131131
return this;
132132
}
133133

134134
public NormalizedDataMetricsDataBuilder scopeAttributes(List<KeyValue> attributes) {
135135
if (attributes != null) {
136-
scopeAttributes.addAll(attributes);
136+
scopeAttributes.addAll(normalizeAttributeKeys(attributes));
137137
}
138138
return this;
139139
}
@@ -148,6 +148,16 @@ public NormalizedDataMetricsDataBuilder gaugeMetric(NormalizedData normalizedDat
148148
}
149149

150150
public MetricsData build() {
151+
for (ResourceMetrics resourceMetrics : metricsData.getResourceMetrics()) {
152+
if (resourceMetrics.getResource() != null) {
153+
addMissingAttributes(resourceMetrics.getResource().getAttributes(), resourceAttributes);
154+
}
155+
for (ScopeMetrics scopeMetrics : resourceMetrics.getScopeMetrics()) {
156+
if (scopeMetrics.getScope() != null) {
157+
addMissingAttributes(scopeMetrics.getScope().getAttributes(), scopeAttributes);
158+
}
159+
}
160+
}
151161
return metricsData;
152162
}
153163

@@ -226,9 +236,9 @@ private NumberDataPoint dataPoint(NormalizedData normalizedData, String contentP
226236
NumberDataPoint dataPoint = new NumberDataPoint();
227237
dataPoint.setTimeUnixNano(OtlpTimeUtils.unixNanoAsString(normalizedData.getTimeObserved()));
228238
dataPoint.setAsDouble(metricValue(content, jsonValue, jsonType).doubleValue());
229-
dataPoint.getAttributes().add(OtlpAttributeUtils.attribute("iot.content.path", contentPath));
239+
dataPoint.getAttributes().add(attribute("iot.content.path", contentPath));
230240
dataPoint.getAttributes()
231-
.add(OtlpAttributeUtils.attribute("iot.content.type", normalizedData.getContentType()));
241+
.add(attribute("iot.content.type", normalizedData.getContentType()));
232242
return dataPoint;
233243
}
234244

@@ -238,14 +248,25 @@ public String metricName(NormalizedData normalizedData) {
238248

239249
public String metricName(String contentPath) {
240250
if (!hasText(contentPath)) {
241-
return metricPrefix + ".value";
251+
return toDotSeparatedSnakeCase(metricPrefix) + ".value";
242252
}
243-
String sanitizedPath = contentPath.strip().replace('\\', '/').replaceAll("^/+", "").replace('/', '.')
244-
.replaceAll("[^A-Za-z0-9_.-]+", "_").replaceAll("\\.+", ".");
253+
String sanitizedPath = toDotSeparatedSnakeCase(contentPath.strip().replace('\\', '/').replaceAll("^/+", "")
254+
.replace('/', '.').replaceAll("[^A-Za-z0-9_.-]+", "_").replaceAll("\\.+", "."));
245255
if (!hasText(sanitizedPath)) {
246256
sanitizedPath = "value";
247257
}
248-
return metricPrefix + "." + sanitizedPath;
258+
return toDotSeparatedSnakeCase(metricPrefix) + "." + sanitizedPath;
259+
}
260+
261+
public static String toDotSeparatedSnakeCase(String input) {
262+
if (!hasText(input)) {
263+
return input;
264+
}
265+
String[] parts = input.strip().replace('\\', '/').replace('/', '.').split("\\.");
266+
for (int i = 0; i < parts.length; i++) {
267+
parts[i] = toSnakeCase(parts[i]);
268+
}
269+
return String.join(".", parts).replaceAll("\\.+", ".");
249270
}
250271

251272
public BigDecimal metricValue(NormalizedData normalizedData) {
@@ -283,6 +304,19 @@ private BigDecimal metricValue(String content, OracleJsonValue jsonValue, Oracle
283304
}
284305
}
285306

307+
private static String toSnakeCase(String value) {
308+
if (!hasText(value)) {
309+
return value;
310+
}
311+
return value.strip()
312+
.replaceAll("([A-Z]+)([A-Z][a-z])", "$1_$2")
313+
.replaceAll("([a-z0-9])([A-Z])", "$1_$2")
314+
.replaceAll("[^A-Za-z0-9]+", "_")
315+
.replaceAll("_+", "_")
316+
.replaceAll("^_|_$", "")
317+
.toLowerCase(java.util.Locale.ROOT);
318+
}
319+
286320
private static String appendPath(String contentPath, String fieldName) {
287321
if (!hasText(contentPath)) {
288322
return fieldName;
@@ -300,10 +334,8 @@ private static boolean isNumeric(OracleJsonType jsonType) {
300334

301335
private ResourceMetrics resourceMetrics(NormalizedData normalizedData) {
302336
Resource resource = new Resource();
303-
resource.getAttributes().add(OtlpAttributeUtils.attribute("service.name", serviceName));
304-
resource.getAttributes().add(OtlpAttributeUtils.attribute("iot.digital_twin.instance_id",
305-
normalizedData.getDigitalTwinInstanceId()));
306-
resource.getAttributes().addAll(resourceAttributes);
337+
resource.getAttributes().add(attribute("service.name", serviceName));
338+
resource.getAttributes().add(attribute("iot.digital_twin.instance_id", normalizedData.getDigitalTwinInstanceId()));
307339

308340
ResourceMetrics resourceMetrics = new ResourceMetrics();
309341
resourceMetrics.setResource(resource);
@@ -312,15 +344,40 @@ private ResourceMetrics resourceMetrics(NormalizedData normalizedData) {
312344

313345
private ScopeMetrics scopeMetrics() {
314346
InstrumentationScope scope = new InstrumentationScope();
315-
scope.setName(scopeName);
347+
scope.setName(toDotSeparatedSnakeCase(scopeName));
316348
scope.setVersion(scopeVersion);
317-
scope.getAttributes().addAll(scopeAttributes);
318349

319350
ScopeMetrics scopeMetrics = new ScopeMetrics();
320351
scopeMetrics.setScope(scope);
321352
return scopeMetrics;
322353
}
323354

355+
private static void addMissingAttributes(List<KeyValue> target, List<KeyValue> attributes) {
356+
for (KeyValue attribute : attributes) {
357+
if (!target.contains(attribute)) {
358+
target.add(attribute);
359+
}
360+
}
361+
}
362+
363+
private static List<KeyValue> normalizeAttributeKeys(List<KeyValue> attributes) {
364+
List<KeyValue> normalizedAttributes = new ArrayList<>();
365+
for (KeyValue attribute : attributes) {
366+
if (attribute != null) {
367+
normalizedAttributes.add(attribute(attribute.getKey(), attribute.getValue()));
368+
}
369+
}
370+
return normalizedAttributes;
371+
}
372+
373+
private static KeyValue attribute(String key, String value) {
374+
return OtlpAttributeUtils.attribute(toDotSeparatedSnakeCase(key), value);
375+
}
376+
377+
private static KeyValue attribute(String key, AnyValue value) {
378+
return OtlpAttributeUtils.attribute(toDotSeparatedSnakeCase(key), value);
379+
}
380+
324381
private static boolean hasText(String value) {
325382
return value != null && !value.isBlank();
326383
}

0 commit comments

Comments
 (0)