Is your feature request related to a problem? Please describe.
It would be nice to include a feature which reads multiline logs like error stack traces or logs having database queries where each parts of query are present in different lines. I have sample log 2023-06-11T12:17:06,027 INFO [deft-faye-4dd0-Scheduler_Worker-22] {} c.t.w.a.s.j.ExtractTeradataJob$ExtractTask.executeTask(1429) - Executing sql: locking row for access
sel
s.object_id
,cast(null as varchar(128)) as databasename
,cast(null as varchar(128)) as tablename
,s.sum_currentperm
,c.blccompratio
,current_timestamp(6) as thetimestamp
from (
sel
hashrow(t.databasename,t.tablename) as object_id
,t.databasename
,t.tablename
,sum(t.currentperm) as sum_currentperm
from dbc.tablesizev t
qualify row_number() over (order by sum_currentperm desc) <= 100
group by 2,3
) s
left outer join dbc.statsv c
on s.databasename = c.databasename
and s.tablename = c.tablename
and c.statstype = 'T'
and c.validstats = 'T'
and c.columnname is null
and I have use grok pattern %{TIMESTAMP_ISO8601:log_timestamp} %{LOGLEVEL:loglevel} [%{DATA:thread}] {} %{GREEDYDATA:logmessage} to extract the log into different fields but what is happening here is the logmessage field is able to extract the text only till Executing sql: locking row for access so whatever is below this data its not able to extract into logmessage field. I heard that GREEDYSEARCH only extracts data until new line is encountered. Also the query instead of being considered as single log event each new line is treated as new event.
Describe the solution you'd like
I would like to have parameter multiline code , which is present in logstash like
codec:
multiline:
pattern:
what:
negate:
this particular parameters will be able to read multiline logs but in data prepper we don't have that. If there is one could you please let me know the solution. I have attached screenshot which shows how mmultiline logs are being treated
Describe alternatives you've considered (Optional)
A clear and concise description of any alternative solutions or features you've considered.
Additional context
Is your feature request related to a problem? Please describe.
It would be nice to include a feature which reads multiline logs like error stack traces or logs having database queries where each parts of query are present in different lines. I have sample log 2023-06-11T12:17:06,027 INFO [deft-faye-4dd0-Scheduler_Worker-22] {} c.t.w.a.s.j.ExtractTeradataJob$ExtractTask.executeTask(1429) - Executing sql: locking row for access
sel
s.object_id
,cast(null as varchar(128)) as databasename
,cast(null as varchar(128)) as tablename
,s.sum_currentperm
,c.blccompratio
,current_timestamp(6) as thetimestamp
from (
sel
hashrow(t.databasename,t.tablename) as object_id
,t.databasename
,t.tablename
,sum(t.currentperm) as sum_currentperm
from dbc.tablesizev t
qualify row_number() over (order by sum_currentperm desc) <= 100
group by 2,3
) s
left outer join dbc.statsv c
on s.databasename = c.databasename
and s.tablename = c.tablename
and c.statstype = 'T'
and c.validstats = 'T'
and c.columnname is null
and I have use grok pattern %{TIMESTAMP_ISO8601:log_timestamp} %{LOGLEVEL:loglevel} [%{DATA:thread}] {} %{GREEDYDATA:logmessage} to extract the log into different fields but what is happening here is the logmessage field is able to extract the text only till Executing sql: locking row for access so whatever is below this data its not able to extract into logmessage field. I heard that GREEDYSEARCH only extracts data until new line is encountered. Also the query instead of being considered as single log event each new line is treated as new event.
Describe the solution you'd like
I would like to have parameter multiline code , which is present in logstash like
codec:
multiline:
pattern:
what:
negate:
this particular parameters will be able to read multiline logs but in data prepper we don't have that. If there is one could you please let me know the solution. I have attached screenshot which shows how mmultiline logs are being treated
Describe alternatives you've considered (Optional)
A clear and concise description of any alternative solutions or features you've considered.
Additional context
