You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: IoTDBJDBC/README.md
+90-3Lines changed: 90 additions & 3 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -3,8 +3,97 @@ This is built using maven and the micronaut libraries. see below for details on
3
3
4
4
YOU MUST be running this on a compute resource on a private VCN that is connected to the IOT database, see the IOT documentation at https://docs.oracle.com/en-us/iaas/Content/internet-of-things/connect-database.htm for details on how to set this up.
5
5
6
+
## Dynamic source and handler selection
6
7
7
-
]## Micronaut 4.10.10 Documentation
8
+
The application uses Micronaut dependency injection to decide which IoT data sources and output handlers are active at runtime. The code does not use a central switch statement. Instead, each optional reader, filter, processor, and output is a Micronaut `@Singleton` bean guarded with `@Requires` annotations. If the required configuration properties are present, and any `.enabled` property is set to `true`, Micronaut creates that bean. If the properties are missing or disabled, the bean is not created and is not injected anywhere else.
9
+
10
+
### Selecting input sources
11
+
12
+
All database and AQ readers implement `IoTDBClient`. `JDBCRunner` asks Micronaut for `List<IoTDBClient>`, so only the readers whose `@Requires` conditions pass are injected. The runner sorts the injected clients by each client's configured `getOrder()` value, calls `configureDBClient()`, then starts them with `startDBProcessing()`. This means one configuration file can enable one source, several sources, or none.
13
+
14
+
The available input clients are:
15
+
16
+
| Client | Data source | Main enable property | Order property |
17
+
| --- | --- | --- | --- |
18
+
|`IoTJDBCConnectionTestReader`| Reads sample rows from the `raw_data` table over JDBC. |`iotdatacache.jdbc.doconnectiontestread.enabled=true`|`iotdatacache.jdbc.doconnectiontestread.order`|
19
+
|`IoTAQNormalizedDataBatchReader`| Dequeues batches from the normalized data AQ queue and passes each message to the normalized data handler chain. |`iotdatacache.aq.normalizeddata.batchreader.enabled=true`|`iotdatacache.aq.normalizeddata.batchreader.order`|
20
+
|`IoTAQNormalizedDataIndividualReader`| Dequeues normalized AQ messages one at a time and passes them to the normalized data handler chain. |`iotdatacache.aq.normalizeddata.individualreader.enabled=true`|`iotdatacache.aq.normalizeddata.individualreader.order`|
21
+
|`IoTAQRawDataIndividualReader`| Dequeues raw data AQ messages one at a time and passes them to the raw data handler chain. |`iotdatacache.aq.rawdata.individualreader.enabled=true`|`iotdatacache.aq.rawdata.individualreader.order`|
22
+
|`IoTAQNormalizedDataListener`| Registers an AQ notification listener for normalized data. |`iotdatacache.aq.listener.enabled=true`|`iotdatacache.aq.listener.order`|
23
+
24
+
Readers can also have source-specific settings. For example, the batch reader uses `iotdatacache.aq.normalizeddata.batchreader.batchsize`, and the AQ readers use read timeout and subscriber-name properties. See `config/config.properties` for examples of the available settings.
25
+
26
+
The normalized batch, normalized individual, and raw individual AQ readers feed messages into the handler services. The notification listener currently dequeues and logs normalized messages through its core output method.
27
+
28
+
### AQ versus JDBC
29
+
30
+
The JDBC reader and the AQ readers both use database connections, but they use different database interaction models. The JDBC reader runs SQL directly against database tables, for example reading recent rows from `raw_data`. It is useful for direct queries, connection checks, diagnostics, and any case where the application wants to decide exactly which table rows to fetch.
31
+
32
+
AQ, or Advanced Queuing, treats incoming IoT data as queued messages rather than table rows to query. The AQ readers subscribe to the IoT queue, dequeue messages from it, convert the queue payload into `RawData` or `NormalizedData`, and then pass the converted object to the appropriate message handler service.
33
+
34
+
The current AQ readers can run as polling loops. For example, the individual and batch readers repeatedly call `dequeue` with a configured wait timeout. If no message is available before the timeout, the loop simply waits again. When a message is retrieved, the reader immediately hands it to `RawDataMessageHandlerService` or `NormalizedDataMessageHandlerService`. Because downstream code receives each retrieved message through the handler chain as soon as the polling loop obtains it, the rest of the application can treat the flow as listener-like even though the reader itself is implemented as a polling loop.
35
+
36
+
The AQ `dequeue` call is blocking up to the configured wait timeout. For a single-message dequeue, the call returns as soon as one message is available, or it times out if no message arrives. For a batch dequeue, the call requests up to the configured batch size and returns when the requested number of entries has been retrieved or when the wait timeout expires. That means a batch call may return a full batch, a partial batch, or no data if the queue stays empty until timeout.
37
+
38
+
Requesting one entry at a time keeps per-message latency low and makes error handling simple, because each dequeue result maps directly to one handler-chain invocation. It also commits progress frequently. The tradeoff is that it performs more database round trips when the queue is busy, so throughput can be lower.
39
+
40
+
Requesting multiple entries in one dequeue can improve throughput by reducing database round trips and allowing a batch reader to drain bursts of queued messages more efficiently. The tradeoff is that an early message in the batch may wait until the batch fills or the timeout fires, and processing/commit behavior is grouped around the batch read. Batch reads are usually better for sustained or bursty load; single-message reads are usually better when immediate processing and simpler operational behavior matter more than maximum throughput.
41
+
42
+
This gives the application two useful modes. Direct JDBC access is table/query oriented. AQ access is message/stream oriented and is better suited to continuous processing, filtering, transformation, and output handling.
43
+
44
+
### Selecting filters, processors, and outputs
45
+
46
+
Raw and normalized messages are handled by separate chains:
47
+
48
+
-`RawDataMessageHandlerService` receives a Micronaut-injected `List<RawDataMessageHandler>`.
49
+
-`NormalizedDataMessageHandlerService` receives a Micronaut-injected `List<NormalizedDataMessageHandler>`.
50
+
51
+
As with input clients, only handlers whose `@Requires` properties match the runtime configuration are created and injected. Each handler provides an order value from configuration, and the service sorts the injected handlers before processing messages. A handler returns an array of messages. Returning one or more messages passes those messages to the next handler in the chain; returning an empty array stops that branch. This allows the same mechanism to support filters, test processors, and final output handlers.
52
+
53
+
The raw data handler chain can include filters such as:
- text output using `messagehandler.output.rawdata.textoutput.*`
62
+
- HTTP output enabled with `messagehandler.output.rawdata.httpclient.enabled`; the current code reads its order and type from `messagehandler.output.rawdata.httpclient.enabled.order` and `messagehandler.output.rawdata.httpclient.enabled.type`
63
+
- NoSQL output settings under `messagehandler.output.rawdata.nosql.*`; the current code enables this bean with `messagehandler.filter.rawdata.nosql.enabled`
64
+
65
+
The normalized data handler chain can include filters and test processors such as:
The usual pattern is to set a handler's `.enabled` property to `true` and provide its `.order` property. Lower order values run earlier. Filters normally sit before outputs, and outputs can either pass the message through to later handlers or terminate that branch by returning no messages.
In this example, Micronaut creates the raw AQ reader, the content-type filter, and the text output handler. The reader receives raw AQ messages, the raw handler service runs the content-type filter first, and matching messages are then passed to the text output handler.
Copy file name to clipboardExpand all lines: IoTDBJDBC/src/main/java/com/oracle/demo/timg/iot/iotdbjdbc/messagehandler/filters/DeviceModelMessageFilterCore.java
Copy file name to clipboardExpand all lines: IoTDBJDBC/src/main/java/com/oracle/demo/timg/iot/iotdbjdbc/messagehandler/outputs/http/IoTOutputHttpClientRequestFilter.java
+3-3Lines changed: 3 additions & 3 deletions
Original file line number
Diff line number
Diff line change
@@ -49,12 +49,12 @@ Software and the Larger Work(s), and to sublicense the foregoing rights on
Copy file name to clipboardExpand all lines: IoTDBJDBC/src/main/java/com/oracle/demo/timg/iot/iotdbjdbc/messagehandler/outputs/http/IoTOutputHttpClientSettings.java
+2-4Lines changed: 2 additions & 4 deletions
Original file line number
Diff line number
Diff line change
@@ -43,10 +43,8 @@ Software and the Larger Work(s), and to sublicense the foregoing rights on
0 commit comments