Updated functionality so that sourceType now requires payloadKey to be defined. Docs updated. Added api validation, e2e and functional tests.

Author: marpears

api/observability/v1/output_types.go

@@ -1251,6 +1251,7 @@ type SplunkAuthentication struct {
 
 // Splunk Deliver log data to Splunk’s HTTP Event Collector
 // Provides optional extra properties for `type: splunk_hec` ('splunk_hec_logs' after Vector 0.23
+// +kubebuilder:validation:XValidation:rule="!has(self.sourceType) || has(self.payloadKey)",message="sourceType can only be set when payloadKey is defined"
 type Splunk struct {
 	// Authentication sets credentials for authenticating the requests.
 	//
@@ -1320,19 +1321,30 @@ type Splunk struct {
 	// +operator-sdk:csv:customresourcedefinitions:type=spec,displayName="Source",xDescriptors={"urn:alm:descriptor:com.tectonic.ui:text"}
 	Source string `json:"source,omitempty"`
 
-	// SourceType can be used to specify a pretrained or custom sourcetype in Splunk.
-	// If not specified, the Splunk source type used is `_json` but may be overridden when using `payloadKey`, depending on the structure of the final event payload.
+	// SourceType can be used to specify a pretrained or custom source type in Splunk, but can only be set when PayloadKey is defined.
+	//
+	// WARNING: The administrator is responsible for configuring the pipeline so the source type matches the log entry. The collector makes no effort or validation to ensure they match.
+	//
+	// If SourceType is not specified, the source type used is `_json`. If using PayloadKey without SourceType, the source type used will be either `_json` or `generic_single_line`, depending on the structure of the final event payload.
 	// Details in: docs/features/logforwarding/outputs/splunk-forwarding.adoc
 	//
+	// The SourceType can be a combination of static and dynamic values consisting of field paths followed by `||` followed by another field path or a static value.
+	// A dynamic value is encased in single curly brackets `{}` and MUST end with a static fallback value separated with `||`.
+	//
+	// Static values can only contain alphanumeric characters along with dashes, underscores, dots and forward slashes.
+	//
 	// Examples:
 	//
-	//  1. "access_combined"
+	//  1. "log4j"
 	//
-	//  2. "log4j"
+	//  2. foo-{.bar||"none"}
 	//
-	//  3. "my:custom:sourcetype"
+	//  3. {.foo||.bar||"missing"}
+	//
+	//  4. foo.{.bar.baz||.qux.quux.corge||.grault||"nil"}-waldo.fred{.plugh||"none"}
 	//
 	// +kubebuilder:validation:Optional
+	// +kubebuilder:validation:Pattern:=`^(([a-zA-Z0-9-_.\/])*(\{(\.[a-zA-Z0-9_]+|\."[^"]+")+((\|\|)(\.[a-zA-Z0-9_]+|\.?"[^"]+")+)*\|\|"[^"]*"\})*)*$`
 	// +operator-sdk:csv:customresourcedefinitions:type=spec,displayName="SourceType",xDescriptors={"urn:alm:descriptor:com.tectonic.ui:text"}
 	SourceType string `json:"sourceType,omitempty"`
 

bundle/manifests/cluster-logging.clusterserviceversion.yaml

@@ -1917,17 +1917,27 @@ spec:
         x-descriptors:
         - urn:alm:descriptor:com.tectonic.ui:text
       - description: |-
-          SourceType can be used to specify a pretrained or custom sourcetype in Splunk.
-          If not specified, the Splunk source type used is `_json` but may be overridden when using `payloadKey`, depending on the structure of the final event payload.
+          SourceType can be used to specify a pretrained or custom source type in Splunk, but can only be set when PayloadKey is defined.
+
+          WARNING: The administrator is responsible for configuring the pipeline so the source type matches the log entry. The collector makes no effort or validation to ensure they match.
+
+          If SourceType is not specified, the source type used is `_json`. If using PayloadKey without SourceType, the source type used will be either `_json` or `generic_single_line`, depending on the structure of the final event payload.
           Details in: docs/features/logforwarding/outputs/splunk-forwarding.adoc
 
+          The SourceType can be a combination of static and dynamic values consisting of field paths followed by `||` followed by another field path or a static value.
+          A dynamic value is encased in single curly brackets `{}` and MUST end with a static fallback value separated with `||`.
+
+          Static values can only contain alphanumeric characters along with dashes, underscores, dots and forward slashes.
+
           Examples:
 
-           1. "access_combined"
+           1. "log4j"
+
+           2. foo-{.bar||"none"}
 
-           2. "log4j"
+           3. {.foo||.bar||"missing"}
 
-           3. "my:custom:sourcetype"
+           4. foo.{.bar.baz||.qux.quux.corge||.grault||"nil"}-waldo.fred{.plugh||"none"}
         displayName: SourceType
         path: outputs[0].splunk.sourceType
         x-descriptors:

bundle/manifests/observability.openshift.io_clusterlogforwarders.yaml

@@ -3855,17 +3855,28 @@ spec:
                           type: string
                         sourceType:
                           description: |-
-                            SourceType can be used to specify a pretrained or custom sourcetype in Splunk.
-                            If not specified, the Splunk source type used is `_json` but may be overridden when using `payloadKey`, depending on the structure of the final event payload.
+                            SourceType can be used to specify a pretrained or custom source type in Splunk, but can only be set when PayloadKey is defined.
+
+                            WARNING: The administrator is responsible for configuring the pipeline so the source type matches the log entry. The collector makes no effort or validation to ensure they match.
+
+                            If SourceType is not specified, the source type used is `_json`. If using PayloadKey without SourceType, the source type used will be either `_json` or `generic_single_line`, depending on the structure of the final event payload.
                             Details in: docs/features/logforwarding/outputs/splunk-forwarding.adoc
 
+                            The SourceType can be a combination of static and dynamic values consisting of field paths followed by `||` followed by another field path or a static value.
+                            A dynamic value is encased in single curly brackets `{}` and MUST end with a static fallback value separated with `||`.
+
+                            Static values can only contain alphanumeric characters along with dashes, underscores, dots and forward slashes.
+
                             Examples:
 
-                             1. "access_combined"
+                             1. "log4j"
+
+                             2. foo-{.bar||"none"}
 
-                             2. "log4j"
+                             3. {.foo||.bar||"missing"}
 
-                             3. "my:custom:sourcetype"
+                             4. foo.{.bar.baz||.qux.quux.corge||.grault||"nil"}-waldo.fred{.plugh||"none"}
+                          pattern: ^(([a-zA-Z0-9-_.\/])*(\{(\.[a-zA-Z0-9_]+|\."[^"]+")+((\|\|)(\.[a-zA-Z0-9_]+|\.?"[^"]+")+)*\|\|"[^"]*"\})*)*$
                           type: string
                         tuning:
                           description: Tuning specs tuning for the output
@@ -3920,6 +3931,9 @@ spec:
                       - authentication
                       - url
                       type: object
+                      x-kubernetes-validations:
+                      - message: sourceType can only be set when payloadKey is defined
+                        rule: '!has(self.sourceType) || has(self.payloadKey)'
                     syslog:
                       description: Syslog configures forwarding log events to a receiver
                         using the syslog protocol

config/crd/bases/observability.openshift.io_clusterlogforwarders.yaml

@@ -3855,17 +3855,28 @@ spec:
                           type: string
                         sourceType:
                           description: |-
-                            SourceType can be used to specify a pretrained or custom sourcetype in Splunk.
-                            If not specified, the Splunk source type used is `_json` but may be overridden when using `payloadKey`, depending on the structure of the final event payload.
+                            SourceType can be used to specify a pretrained or custom source type in Splunk, but can only be set when PayloadKey is defined.
+
+                            WARNING: The administrator is responsible for configuring the pipeline so the source type matches the log entry. The collector makes no effort or validation to ensure they match.
+
+                            If SourceType is not specified, the source type used is `_json`. If using PayloadKey without SourceType, the source type used will be either `_json` or `generic_single_line`, depending on the structure of the final event payload.
                             Details in: docs/features/logforwarding/outputs/splunk-forwarding.adoc
 
+                            The SourceType can be a combination of static and dynamic values consisting of field paths followed by `||` followed by another field path or a static value.
+                            A dynamic value is encased in single curly brackets `{}` and MUST end with a static fallback value separated with `||`.
+
+                            Static values can only contain alphanumeric characters along with dashes, underscores, dots and forward slashes.
+
                             Examples:
 
-                             1. "access_combined"
+                             1. "log4j"
+
+                             2. foo-{.bar||"none"}
 
-                             2. "log4j"
+                             3. {.foo||.bar||"missing"}
 
-                             3. "my:custom:sourcetype"
+                             4. foo.{.bar.baz||.qux.quux.corge||.grault||"nil"}-waldo.fred{.plugh||"none"}
+                          pattern: ^(([a-zA-Z0-9-_.\/])*(\{(\.[a-zA-Z0-9_]+|\."[^"]+")+((\|\|)(\.[a-zA-Z0-9_]+|\.?"[^"]+")+)*\|\|"[^"]*"\})*)*$
                           type: string
                         tuning:
                           description: Tuning specs tuning for the output
@@ -3920,6 +3931,9 @@ spec:
                       - authentication
                       - url
                       type: object
+                      x-kubernetes-validations:
+                      - message: sourceType can only be set when payloadKey is defined
+                        rule: '!has(self.sourceType) || has(self.payloadKey)'
                     syslog:
                       description: Syslog configures forwarding log events to a receiver
                         using the syslog protocol

config/manifests/bases/cluster-logging.clusterserviceversion.yaml

@@ -1840,17 +1840,27 @@ spec:
         x-descriptors:
         - urn:alm:descriptor:com.tectonic.ui:text
       - description: |-
-          SourceType can be used to specify a pretrained or custom sourcetype in Splunk.
-          If not specified, the Splunk source type used is `_json` but may be overridden when using `payloadKey`, depending on the structure of the final event payload.
+          SourceType can be used to specify a pretrained or custom source type in Splunk, but can only be set when PayloadKey is defined.
+
+          WARNING: The administrator is responsible for configuring the pipeline so the source type matches the log entry. The collector makes no effort or validation to ensure they match.
+
+          If SourceType is not specified, the source type used is `_json`. If using PayloadKey without SourceType, the source type used will be either `_json` or `generic_single_line`, depending on the structure of the final event payload.
           Details in: docs/features/logforwarding/outputs/splunk-forwarding.adoc
 
+          The SourceType can be a combination of static and dynamic values consisting of field paths followed by `||` followed by another field path or a static value.
+          A dynamic value is encased in single curly brackets `{}` and MUST end with a static fallback value separated with `||`.
+
+          Static values can only contain alphanumeric characters along with dashes, underscores, dots and forward slashes.
+
           Examples:
 
-           1. "access_combined"
+           1. "log4j"
+
+           2. foo-{.bar||"none"}
 
-           2. "log4j"
+           3. {.foo||.bar||"missing"}
 
-           3. "my:custom:sourcetype"
+           4. foo.{.bar.baz||.qux.quux.corge||.grault||"nil"}-waldo.fred{.plugh||"none"}
         displayName: SourceType
         path: outputs[0].splunk.sourceType
         x-descriptors:

docs/features/logforwarding/outputs/splunk-forwarding.adoc

@@ -54,7 +54,7 @@ spec:
 2. `url`: The base URL of the Splunk instance.
 3. `index`: Optional. The name of the index to send events to. If not specified, the default index defined within Splunk is used. This supports template syntax to allow dynamic per-event values.
 4. `source`: Optional. The source of events sent to this sink. This supports template syntax to allow dynamic per-event values.
-5. `sourceType`: Optional. Can be used to specify a pretrained or custom sourcetype in Splunk. If not specified it will default to `_json` and be detected by .payloadkey.
+5. `sourceType`: Optional. Can be used to specify a https://help.splunk.com/en/splunk-enterprise/get-started/get-data-in/10.2/configure-source-types/list-of-pretrained-source-types[pretrained] or custom sourcetype in Splunk when the payloadKey is additionally defined. If not specified, it will default to `_json`.
 6. `indexedFields`: Optional. Fields to be added to Splunk index.
 7. `payloadKey`: Optional. Specifies record field to use as payload.
 8. `compression`: Optional. Compression configuration, available to are: `none`, `gzip`. Default is `none`.
@@ -67,88 +67,76 @@ following Cluster Log Forwarder's conventions.
 === `sourceType`
 In Splunk, the source type is used to define how incoming data should be parsed, interpreted, and categorised. Splunk includes many built-in source types and also allows for custom source types to be defined. The `sourceType` field should reference a source type configured in Splunk.
 
-If not specified, the Splunk source type used is `_json` but may be overridden when using `payloadKey`, depending on the structure of the final event payload.
+This field must be used in combination with `payloadKey`.
 
-An example of a use case for this field is where a custom Splunk source type has been created to interpret particular log message content where key/value pairs are extracted and transformed into Splunk fields for use with dashboards.
+If `sourceType` is not specified, the source type used is `_json`. If using `payloadKey` without `sourceType`, the source type used will be either `_json` or `generic_single_line`, depending on the structure of the final event payload.
 
-Here is an example of a CLF where log events for an app are forwarded to a Splunk output sink with a user-defined source type :
+WARNING: The administrator is responsible for configuring the pipeline so the source type matches the log entry. The collector makes no effort or validation to ensure they match.
 
-```
+The `sourceType` field can be defined in the CLF as a templated field to get its value from the stuctured log event, such as a pod label. 
+
+Below is an example of a pod which defines the source type to use in a label named `splunk_sourcetype`:
+
+[source,yaml]
+----
+apiVersion: v1
+kind: Pod
+metadata:
+  name: myapp
+  labels:
+    app: myapp
+    splunk_sourcetype: log4j
 spec:
-  inputs:
-    - application:
-        includes:
-          - container: myapp
-            namespace: my-app-*
-      name: myapp
-      type: application
-  outputs:
-    - name: splunk-myapp
-      splunk:
-        authentication:
-          token:
-            key: hecToken
-            secretName: splunk-myapp
-        sourceType: 'myapp:sourcetype'
-        url: 'https://splunk.customer.com:8088'
-      type: splunk
-  pipelines:
+  containers:
     - name: myapp
-      inputRefs:
-        - myapp
-      outputRefs:
-        - splunk-myapp
-```
-
-If a source type needs to be referenced for a particular application's log events, but is not appropriate for others, be sure to structure the ClusterLogFowarder so that it does not misrepresent the format of log events by using a single Splunk output sink.
+      image: myapp:latest
+      ports:
+        - containerPort: 80
+----
 
-Here is an example of a CLF where cluster-wide general app log events are forwarded to a Splunk output sink using the default source type of `_json`, and log events for an app are forwarded to a different Splunk output sink with a user-defined source type :
+Below is an example of the CLF which sets `payloadKey` to `.message` and uses a template to set the `sourceType` to match the `splunk_sourcetype` pod label. If the label does not exist, the source type of "generic_single_line" will be used, which is a built-in Splunk source type.
 
-```
+[source,yaml]
+----
+apiVersion: observability.openshift.io/v1
+kind: ClusterLogForwarder
+metadata:
+  name: collector
+  namespace: openshift-logging
 spec:
+  collector:
+    resources:
+      limits:
+        cpu: "6"
+        memory: 2Gi
+      requests:
+        cpu: 500m
+        memory: 64Mi
   inputs:
-    - name: application-general
-      application:
-        excludes:
-          - container: myapp
-            namespace: my-app-*
-      type: application
-    - name: application-myapp
-      application:
-        includes:
-          - container: myapp
-            namespace: my-app-*
-      type: application
+  - application: {}
+    name: application
+    type: application
+  managementState: Managed
   outputs:
-    - name: splunk-application-general
-      splunk:
-        authentication:
-          token:
-            key: hecToken
-            secretName: splunk-application-general
-        url: 'https://splunk.customer.com:8088'
-      type: splunk
-    - name: splunk-application-myapp
-      splunk:
-        authentication:
-          token:
-            key: hecToken
-            secretName: splunk-application-myapp
-        sourceType: 'myapp:sourcetype'
-        url: 'https://splunk.customer.com:8088'
-      type: splunk
+  - name: splunk
+    splunk:
+      authentication:
+        token:
+          key: hecToken
+          secretName: splunk-secret
+      payloadKey: .message
+      sourceType: '{.kubernetes.labels.splunk_sourcetype||"generic_single_line"}'
+      url: http://splunk.customer.com:8088
+    type: splunk
   pipelines:
-    - name: splunk-application-general
-      inputRefs:
-        - application-general
-      outputRefs:
-        - splunk-application-general
-    - name: splunk-application-myapp
-      inputRefs:
-        - application-myapp
-      outputRefs:
-        - splunk-application-myapp
-```
+  - inputRefs:
+    - application
+    name: forward-pipeline
+    outputRefs:
+    - splunk
+  serviceAccount:
+    name: logging-admin
+----
 
 === `indexedFields`
 
@@ -252,17 +240,16 @@ Transformed field:
 
 === `payloadKey`
 
-By default, `payloadKey` is not set, which means the complete log record is forwarded as the payload.
+By default, `payloadKey` is not set, which means the complete log event payload is forwarded. This field can be used to narrow down the final payload to a specific field within the log event. For example, for the payload to contain only the log message, payloadKey should be set to `.message`.
 
-If `sourceType` is not defined, payloadKey can override the default `sourcetype` value of `_json` :
+Unless `sourceType` has been defined, the selected `payloadKey` field will be checked and a Splunk source type will be nominated :
 
-* `sourceType`:
+* Splunk source type:
 ** `_json` — used when `payloadKey` points to an object.
 ** `generic_single_line` — used when the payload is a primitive value (e.g., string, number, boolean).
 
 NOTE: Use `payloadKey` carefully. Selecting a single field as the payload may cause other important information in the log to be dropped, potentially leading to inconsistent or incomplete log events.
 
-
 === `host`
 
 The  `host` field is *not configurable* through the Cluster Log Forwarder API.