IMAP protocol parser, logger and sticky buffers v2 by glongo · Pull Request #14822 · OISF/suricata

glongo · 2026-02-16T13:14:48Z

Link to ticket: https://redmine.openinfosecfoundation.org/issues/8276

Previous PR: #14792

Describe changes:

Fix body parsing for the Content-Disposition header
Fix body parsing when the TCP payload is reassembled

This introduces a parser for IMAP protocol. Ticket OISF#8276

This introduces a logger for IMAP protocol. Ticket OISF#8276

This implement the following sticky buffers for IMAP protocol: - imap.request - imap.response - imap.email.direction - imap.email.header - imap.email.header.name - imap.email.header.value - imap.email.body Ticket OISF#8276

github-actions · 2026-02-16T13:15:20Z

NOTE: This PR may contain new authors.

catenacyber · 2026-02-16T14:22:38Z

+Email headers are normalized before matching:
+
+- **Header names** are converted to lowercase with hyphens replaced by underscores
+  (e.g. ``Content-Type`` becomes ``content_type``).


Because we do that for smtp.

How so ? Where do you see this in the code ?

See output-json-email-common.c:

struct { const char *config_field; const char *email_field; uint32_t flags; } email_fields[] = { { "reply_to", "reply-to", LOG_EMAIL_DEFAULT }, { "bcc", "bcc", LOG_EMAIL_COMMA|LOG_EMAIL_EXTENDED }, { "message_id", "message-id", LOG_EMAIL_EXTENDED }, { "subject", "subject", LOG_EMAIL_EXTENDED }, { "x_mailer", "x-mailer", LOG_EMAIL_EXTENDED }, { "user_agent", "user-agent", LOG_EMAIL_EXTENDED }, { "received", "received", LOG_EMAIL_ARRAY }, { "x_originating_ip", "x-originating-ip", LOG_EMAIL_DEFAULT }, { "in_reply_to", "in-reply-to", LOG_EMAIL_DEFAULT }, { "references", "references", LOG_EMAIL_DEFAULT }, { "importance", "importance", LOG_EMAIL_DEFAULT }, { "priority", "priority", LOG_EMAIL_DEFAULT }, { "sensitivity", "sensitivity", LOG_EMAIL_DEFAULT }, { "organization", "organization", LOG_EMAIL_DEFAULT }, { "content_md5", "content-md5", LOG_EMAIL_DEFAULT }, { "date", "date", LOG_EMAIL_DEFAULT }, { NULL, NULL, LOG_EMAIL_DEFAULT}, };

Also, @jasonish asked me to do this for compatibility with elasticsearch.

This is for output, not for detection, right ?

I think Philippe is correct in this interpretation.

catenacyber · 2026-02-16T14:23:34Z

   quic-keywords
   nfs-keywords
   smtp-keywords
+   imap-keywords


I think there is more doc than keywords, there is a section where we show an example json output of an imap event

So, the idea would be to also add an output examplo to the https://docs.suricata.io/en/latest/output/eve/index.html chapter.

catenacyber · 2026-02-16T14:25:41Z

    JsonArpLogRegister();
+    /* IMAP JSON logger. */
+    OutputRegisterTxSubModule(LOGGER_JSON_TX, "eve-log", "JsonImapLog", "eve-log.imap",
+            OutputJsonLogInitSub, ALPROTO_IMAP, JsonGenericDirFlowLogger, JsonLogThreadInit,


I wonder if JsonGenericDirFlowLogger is the best for the cases where we see files/emails in both directions...

I've added a direction field to the email object that indicates whether it's to_server or to_client.

Yes, I am not sure I like it

Can you elaborate on what you don't like, exactly?

This, I imagine: https://github.com/OISF/suricata-verify/pull/2908/changes#r2812655828

catenacyber · 2026-02-16T14:28:01Z

+}
+
+fn parse_command_keyword(i: &[u8]) -> IResult<&[u8], ImapCommand> {
+    alt((


why alt of alt ?

This looks inefficient compared to a switch over strings with rust being clever

Bcause alt accepts a maximum of 21 parsers.

Yes, that means alt is not the right way to do this

catenacyber · 2026-02-16T14:28:32Z

+fn parse_quoted_string(i: &[u8]) -> IResult<&[u8], &[u8]> {
+    delimited(
+        char('"'),
+        take_while(|c| c != b'"' && c != b'\r' && c != b'\n'),


Can we not escape quotes in a quoted string ?

catenacyber · 2026-02-16T14:30:22Z

+                    "type": "array",
+                    "description": "Array of IMAP response lines in this transaction",
+                    "items": {
+                        "type": "string"


you can add the suricata section with the keywords ;-)

catenacyber · 2026-02-16T14:31:08Z

+    pub requests: Vec<ImapMessage>,
+    pub responses: Vec<ImapMessage>,
+    pub request_lines: Vec<Vec<u8>>,
+    pub response_lines: Vec<Vec<u8>>,


Do you bound the number of response lines somehow ?

You should :-p https://redmine.openinfosecfoundation.org/issues/8212

catenacyber · 2026-02-16T14:32:32Z

+        if SCAppLayerParserConfParserEnabled(ip_proto_str.as_ptr(), parser.name) != 0 {
+            let _ = AppLayerRegisterParser(&parser, alproto);
+        }
+        if let Some(val) = conf_get("app-layer.protocols.imap.max-tx") {


Can we have pipelined transactions or such ?

catenacyber · 2026-02-16T14:33:26Z

+    if SCAppLayerProtoDetectConfProtoDetectionEnabled(ip_proto_str.as_ptr(), parser.name) != 0 {
+        let alproto = applayer_register_protocol_detection(&parser, 1);
+        ALPROTO_IMAP = alproto;
+        if SCAppLayerParserConfParserEnabled(ip_proto_str.as_ptr(), parser.name) != 0 {


We do not have port-indpendent detection with CAPABILITY pattern anymore ?

Yes we do now and you are removing it... Was this intended ?

catenacyber

Awesome big work

I think there are quite some things to complete

My main remarks would be

how much verbose/configurable do we want the logs to be ?
can we have the existing keywords like email.from work also with IMAP ?

suricata-qa · 2026-02-16T21:05:02Z

Information: QA ran without warnings.

Pipeline = 29614

jufajardini

Left some inline comments, sharing review now to allow for the new iteration to happen!

jufajardini · 2026-02-24T15:49:06Z

        content-inspect-window: 4096
    imap:
-      enabled: detection-only
+      enabled: yes


minor, but: is the plan to have it enabled by default? If so, then I agree that output verbosity is something to consider carefully.

jufajardini · 2026-02-24T15:49:34Z

 - Encrypted quic traffic bypass is now independently controlled through
  ``app-layer.protocols.quic.encryption-handling`` setting. The setting can either
  be ``bypass``, ``track-only`` or ``full``.
+- IMAP parser, logger and sticky buffers have been introduced.


If it will be enabled by default, I think it is worth it mentioning that here.

In general this is not a changelog, so it should only mention breaking changes or other things ppl should really be aware of.

jufajardini · 2026-02-24T20:56:55Z

    JsonArpLogRegister();
+    /* IMAP JSON logger. */
+    OutputRegisterTxSubModule(LOGGER_JSON_TX, "eve-log", "JsonImapLog", "eve-log.imap",
+            OutputJsonLogInitSub, ALPROTO_IMAP, JsonGenericDirFlowLogger, JsonLogThreadInit,


Can you elaborate on what you don't like, exactly?

jufajardini · 2026-02-24T20:59:37Z

+.. table:: **Direction values for imap.email.direction**
+
+        =====  ===========
+        Value  Name
+        =====  ===========
+        0      to_server
+        1      to_client
+        =====  ===========
+


If we want users to use to_server and to_client, wouldn't it be simpler to remove this table?

jufajardini · 2026-02-24T21:01:17Z

+Email headers are normalized before matching:
+
+- **Header names** are converted to lowercase with hyphens replaced by underscores
+  (e.g. ``Content-Type`` becomes ``content_type``).


I think Philippe is correct in this interpretation.

jufajardini · 2026-02-24T21:16:45Z

    JsonArpLogRegister();
+    /* IMAP JSON logger. */
+    OutputRegisterTxSubModule(LOGGER_JSON_TX, "eve-log", "JsonImapLog", "eve-log.imap",
+            OutputJsonLogInitSub, ALPROTO_IMAP, JsonGenericDirFlowLogger, JsonLogThreadInit,


This, I imagine: https://github.com/OISF/suricata-verify/pull/2908/changes#r2812655828

jufajardini · 2026-02-24T21:38:25Z

   quic-keywords
   nfs-keywords
   smtp-keywords
+   imap-keywords


So, the idea would be to also add an output examplo to the https://docs.suricata.io/en/latest/output/eve/index.html chapter.

glongo · 2026-03-09T11:51:30Z

Replaced with #14995

glongo added 4 commits February 16, 2026 12:47

imap: implement parser

1834d1c

This introduces a parser for IMAP protocol. Ticket OISF#8276

imap: implement logger

e063f21

This introduces a logger for IMAP protocol. Ticket OISF#8276

imap: implement sticky buffers

ab5aabd

This implement the following sticky buffers for IMAP protocol: - imap.request - imap.response - imap.email.direction - imap.email.header - imap.email.header.name - imap.email.header.value - imap.email.body Ticket OISF#8276

doc: add imap parser and sticky buffers

5ea4d34

glongo requested review from a team, jasonish, jufajardini and victorjulien as code owners February 16, 2026 13:14

glongo mentioned this pull request Feb 16, 2026

IMAP protocol parser, logger and sticky buffers #14792

Closed

catenacyber reviewed Feb 16, 2026

View reviewed changes

catenacyber requested changes Feb 16, 2026

View reviewed changes

jufajardini added the needs rebase Needs rebase to main label Feb 24, 2026

jufajardini reviewed Feb 24, 2026

View reviewed changes

glongo mentioned this pull request Mar 9, 2026

IMAP protocol parser, logger and sticky buffers v3 #14995

Closed

glongo closed this Mar 9, 2026

Conversation

glongo commented Feb 16, 2026

Uh oh!

github-actions Bot commented Feb 16, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

catenacyber left a comment

Choose a reason for hiding this comment

Uh oh!

suricata-qa commented Feb 16, 2026

Uh oh!

jufajardini left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

glongo commented Mar 9, 2026