Is your feature request related to a problem? Please describe.
In #6249, we added support for the OpenSearch sink to automatically detect if an index is a data stream.
In some cases, users are seeing inconsistencies. This could be because the permissions to detect are not present, or if the index is not always a data stream (perhaps on other clusters)
Describe the solution you'd like
Provide a new sink configuration to control data streams named data_stream.
The possible values are: automatic (the default), always, never.
opensearch:
index: "my-index"
data_stream: "automatic" # or "always" / "never"
The enum would look something like this:
public enum DataStreamStatus {
AUTOMATIC("automatic"),
ALWAYS("always"),
NEVER("never");
}
Describe alternatives you've considered (Optional)
N/A
Additional context
Issues: #2037, #2038
PR: #6249
Is your feature request related to a problem? Please describe.
In #6249, we added support for the OpenSearch sink to automatically detect if an index is a data stream.
In some cases, users are seeing inconsistencies. This could be because the permissions to detect are not present, or if the index is not always a data stream (perhaps on other clusters)
Describe the solution you'd like
Provide a new sink configuration to control data streams named
data_stream.The possible values are:
automatic(the default),always,never.The enum would look something like this:
Describe alternatives you've considered (Optional)
N/A
Additional context
Issues: #2037, #2038
PR: #6249