Stackdriver special fields

When the google-logging-agent receives a structured log record, it treats some fields specially, allowing users to set specific fields in the LogEntry object that get written to the Logging API.

LogEntry fields

Fluent Bit supports some special fields for setting fields on the LogEntry object:

JSON log field	LogEntry field	Type	Description
`logging.googleapis.com/logName`	`logName`	`string`	The log name to write this log to.
`logging.googleapis.com/labels`	`labels`	`object<string, string>`	The labels for this log.
`logging.googleapis.com/severity`	`severity`	`LogSeverity` Enum	The severity of this log.
`logging.googleapis.com/monitored_resource`	`resource`	`MonitoredResource` (without `type`)	Resource labels for this log.
`logging.googleapis.com/operation`	`operation`	`LogEntryOperation`	Additional information about a potentially long-running operation.
`logging.googleapis.com/insertId`	`insertId`	`string`	A unique identifier for the log entry. It's used to order `logEntries`.
`logging.googleapis.com/sourceLocation`	`sourceLocation`	`LogEntrySourceLocation`	Additional information about the source code location that produced the log entry.
`logging.googleapis.com/http_request`	`httpRequest`	`HttpRequest`	A common proto for logging HTTP requests.
`logging.googleapis.com/trace`	`trace`	`string`	Resource name of the trace associated with the log entry.
`logging.googleapis.com/traceSampled`	`traceSampled`	boolean	The sampling decision associated with this log entry.
`logging.googleapis.com/spanId`	`spanId`	`string`	The ID of the trace span associated with this log entry.
`timestamp`	`timestamp`	`object` (protobuf `Timestamp` object format)	An object including the seconds and nanoseconds fields that represent the time.
`timestampSecond` and `timestampNanos`	`timestamp`	`int`	The seconds and nanoseconds that represent the time.

Other special fields

JSON log field	Description
`logging.googleapis.com/projectId`	Changes the project ID that this log will be written to. Ensure that you are authenticated to write logs to this project.
`logging.googleapis.com/local_resource_id`	Overrides the configured `local_resource_id`.

Use special fields

To use a special field, you must add a field with the right name and value to your log. Given an example structured log (internally in MessagePack but shown in JSON for demonstration):

{
  "log": "Hello world!"
}

To use the logging.googleapis.com/logName special field, add it to your structured log as follows:

{
  "log": "Hello world!",
  "logging.googleapis.com/logName": "my_log"
}

For the special fields that map to LogEntry prototypes, add them as objects with field names that match the proto. For example, to use the logging.googleapis.com/operation:

{
  "log": "Hello world!",
  "logging.googleapis.com/operation": {
    "id": "test_id",
    "producer": "test_producer",
    "first": true,
    "last": true
  }
}

Adding special fields to logs is best done through the modify filter for basic fields, or a Lua script using the lua filter for more complex fields.

Basic type special fields

Special fields with basic types (except for the logging.googleapis.com/insertId field) will follow this pattern (demonstrated with the logging.googleapis.com/logName field):

If the special field matches the type, it will be moved to the corresponding LogEntry field. For example:

{
  ...
  "logging.googleapis.com/logName": "my_log"
  ...
}

the logEntry will be:

{
  "jsonPayload": {
    ...
  }
  "logName": "my_log"
  ...
}

If the field is non-empty but invalid, it will be left in the jsonPayload. For example:
```
{
  ...
  "logging.googleapis.com/logName": 12345
  ...
}
```

the logEntry will be:

{
  "jsonPayload": {
    "logging.googleapis.com/logName": 12345
    ...
  }
}

Exceptions [#exceptions-basic]

Insert ID

If the logging.googleapis.com/insertId field has an invalid type, the log will be rejected by the plugin and not sent to Cloud Logging.

Trace sampled

If autoformat_stackdriver_trace is set to true, the value in the trace field is reformatted to match the Cloud Logging format. The Project ID is detected from the Google Metadata server, configured in the plugin, or provided using a special field.

For example, if autoformat_stackdriver_trace is enabled, this:

{
  ...
  "logging.googleapis.com/projectId": "my-project-id",
  "logging.googleapis.com/trace": "12345"
  ...
}

Will become this:

{
  "jsonPayload": {
    ...
  },
  "projectId": "my-project-id",
  "trace": "/projects/my-project-id/traces/12345"
  ...
}

`timestampSecond` and `timestampNano`

The timestampSecond and timestampNano fields don't map directly to the timestamp field in LogEntry so the parsing behaviour deviates from other special fields. Read more in the Timestamp section.

Proto special fields

For special fields that expect the format of a prototype from the LogEntry (except for the logging.googleapis.com/monitored_resource field) will follow this pattern (demonstrated with the logging.googleapis.com/operation field):

If any sub-fields of the proto are empty or in incorrect type, the plugin will set these sub-fields empty. For example:

{
  "logging.googleapis.com/operation": {
    "id": 123, #incorrect type
    # no producer here
    "first": true,
    "last": true
  }
  ...
}

the logEntry will be:

{
  "jsonPayload": {
    ...
  }
  "operation": {
    "first": true,
    "last": true
  }
  ...
}

If the field itself isn't a map, the plugin will leave this field untouched. For example:

{
  ...
  "logging.googleapis.com/operation": "some string",
  ...
}

the logEntry will be:

{
  "jsonPayload": {
    "logging.googleapis.com/operation": "some string",
    ...
  }
  ...
}

If there are extra sub-fields, the plugin will add the recognized fields to the corresponding field in the LogEntry, and preserve the extra sub-fields in jsonPayload. For example:

{
  ...
  "logging.googleapis.com/operation": {
    "id": "test_id",
    "producer": "test_producer",
    "first": true,
    "last": true,
    "extra1": "some string",
    "extra2": 123,
    "extra3": true
  }
  ...
}

the logEntry will be:

{
  "jsonPayload": {
    "logging.googleapis.com/operation": {
      "extra1": "some string",
      "extra2": 123,
      "extra3": true
    }
    ...
  }
  "operation": {
    "id": "test_id",
    "producer": "test_producer",
    "first": true,
    "last": true
  }
  ...
}

Exceptions [#exceptions-proto]

`MonitoredResource` ID

The logging.googleapis.com/monitored_resource field is parsed in a special way, meaning it has some important exceptions:

The type field from the MonitoredResource proto isn't parsed out of the special field. It's read from the resource plugin configuration option. If it's supplied in the logging.googleapis.com/monitored_resource special field, it won't be recognized.

The labels field is expected to be an object<string, string>. If any fields have a value that's not a string, the value is ignored and not preserved. The plugin logs an error and drops the field.

If no valid labels field is found, or if all entries in the labels object are invalid, the logging.googleapis.com/monitored_resource field is dropped. Fluent Bit then automatically sets resource labels using other available information based on the configured resource type.

Timestamp

Fluent Bit supports the following formats of time-related fields:

timestamp

Log body contains a timestamp field that includes the seconds and nanoseconds fields.
```
{
  "timestamp": {
    "seconds": CURRENT_SECONDS,
    "nanos": CURRENT_NANOS
  }
}
```
timestampSeconds/timestampNanos

Log body contains both the timestampSeconds and timestampNanos fields.
```
{
  "timestampSeconds": CURRENT_SECONDS,
  "timestampNanos": CURRENT_NANOS
}
```

If one of the following JSON timestamp representations is present in a structured record, the plugin collapses them into a single representation in the timestamp field in the LogEntry object.

Without time-related fields, the plugin will set the current time as timestamp.

Format 1

Set the input log as follows:

{
  "timestamp": {
    "seconds": 1596149787,
    "nanos": 12345
  }
  ...
}

the logEntry will be:

{
  "jsonPayload": {
    ...
  }
  "timestamp": "2020-07-30T22:56:27.000012345Z"
  ...
}

Format 2

Set the input log as follows:

{
  "timestampSeconds":1596149787,
  "timestampNanos": 12345
  ...
}

the logEntry will be:

{
  "jsonPayload": {
    ...
  }
  "timestamp": "2020-07-30T22:56:27.000012345Z"
  ...
}

If the timestamp object or the timestampSeconds and timestampNanos fields end up being invalid, they will remain in the jsonPayload untouched.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Stackdriver special fields

LogEntry fields

Other special fields

Use special fields

Basic type special fields

Exceptions [#exceptions-basic]

Insert ID

Trace sampled

`timestampSecond` and `timestampNano`

Proto special fields

Exceptions [#exceptions-proto]

`MonitoredResource` ID

Timestamp

Format 1

Format 2

FilesExpand file tree

stackdriver_special_fields.md

Latest commit

History

stackdriver_special_fields.md

File metadata and controls

Stackdriver special fields

LogEntry fields

Other special fields

Use special fields

Basic type special fields

Exceptions [#exceptions-basic]

Insert ID

Trace sampled

timestampSecond and timestampNano

Proto special fields

Exceptions [#exceptions-proto]

MonitoredResource ID

Timestamp

Format 1

Format 2

`timestampSecond` and `timestampNano`

`MonitoredResource` ID