Cherry-pick upstream mailcore2 bug fixes and improvements (#16)

* Cherry-pick upstream mailcore2 bug fixes and improvements

Cherry-picked the following commits from upstream MailCore/mailcore2:

1. fad23d73 - Ensure checking SSL Certificate in IMAP StartTLS
   Security fix: adds certificate validation after StartTLS connection
   to close a potential security gap where certificate validation
   wasn't being performed in that connection mode.

2. cccebc79 - Fix IMAPMessagesRequestKindFullHeaders handling (#1947)
   Bug fix: properly handles the FullHeaders flag when fetching
   messages by UID, fetching complete headers when the flag is set.

3. 29f9488a - Add IMAPMessagesRequestKindAllHeaders flag
   New feature: adds a new flag to fetch all non-parsed headers,
   unlike FullHeaders which fetches a limited set.

These are the only relevant commits since v0.6.4 (Aug 2020) through
Nov 2022 that are bug fixes or functionality improvements for the
C++ core. Excluded: Swift/SPM changes, Android changes, Obj-C specific
fixes, documentation updates, and Xcode project changes.

* Cherry-pick upstream libetpan bug fixes and security patches

This commit cherry-picks 8 critical bug fixes and security patches from
upstream libetpan (versions 1.9.2-1.9.4) to the vendored libetpan 1.9.1.

Patches Applied:

1. CVE-2022-4121 Fix (commit 5c9eb6b)
   - Fixed crash when st_info_list is NULL in mailimap_mailbox_data_status_free
   - Prevents NULL pointer dereference vulnerability
   - File: src/low-level/imap/mailimap_types.c

2. STARTTLS Response Injection Protection - IMAP (commit 1002a01)
   - Detects extra data after STARTTLS response to prevent response injection attacks
   - Returns MAILIMAP_ERROR_STARTTLS if extra data is detected
   - File: src/low-level/imap/mailimap.c

3. STARTTLS Response Injection Protection - SMTP/POP3 (commit 298460a)
   - Same protection for SMTP and POP3 protocols
   - Files: src/low-level/smtp/mailsmtp.c, src/low-level/pop3/mailpop3.c

4. Quota SIGSEGV Fix (commit 180b37a)
   - Fixed SIGSEGV in mailimap_quota_getquotaroot
   - Properly passes parser context to parser functions instead of NULL
   - File: src/low-level/imap/quota_parser.c

5. TLS Timeout Fix for GnuTLS (commit 4aee224)
   - Fixed timeout values for gnutls_handshake_set_timeout (ms vs seconds)
   - Prevents 1000x shorter timeouts than requested on GnuTLS platforms
   - File: src/data-types/mailstream_ssl.c

6. IMAP Logout Return Code Fix (commit 27d6f41)
   - Fixed incorrect error handling when server closes connection after BYE
   - MAILIMAP_ERROR_STREAM is now expected and treated as success
   - File: src/low-level/imap/mailimap.c

7. MIME Field Location Handler (commit b4088cb)
   - Added missing handler for MAILMIME_FIELD_LOCATION in mailmime_write()
   - File: src/low-level/mime/mailmime_write_generic.c

8. snprintf() Output Fix (commit 1fef3a0)
   - Fixed incorrect snprintf() usage in test file
   - File: tests/frm-simple.c

Patches Already Included (in commit 01656c8 from Aug 2022):
- Buffer overwrite fix for empty strings (commit 078b924)
- Memory leak fix in mailimap_hack_date_time_parse (commit 5b43488)
- Proper realloc() usage fix (commit 92f8b23)
- Resent-Bcc header parsing fix (commit 79eefd5)

Patches Excluded:
- Platform-specific commits (iOS/macOS/Android build fixes)
- Build system changes (pkg-config, configure.ac)
- Documentation-only changes
- Reverted commits from upstream
- Features not present in libetpan 1.9.1 (e.g., clientid, TLS SNI)

Analysis Summary:
- Base vendored version: libetpan 1.9.1
- Upstream versions analyzed: 1.9.2, 1.9.3, 1.9.4
- Total commits reviewed: ~100 commits since 1.9.1
- Security fixes applied: 3 (CVE-2022-4121, 2x STARTTLS)
- Bug fixes applied: 5
- Files modified: 8 files, 89 insertions, 24 deletions

This update brings important security fixes and stability improvements to
the email sync engine while maintaining compatibility with the existing
codebase.

* Add vendor library update workflow documentation

Documents the process for cherry-picking upstream bug fixes and
improvements into vendored dependencies (mailcore2, libetpan) while
preserving local modifications.

Includes:
- Step-by-step workflow
- Commit categorization guidelines
- Example from mailcore2 update

---------

Co-authored-by: Claude <noreply@anthropic.com>

Author: bengotow

Vendor/libetpan/src/data-types/mailstream_ssl.c

@@ -624,7 +624,7 @@ static struct mailstream_ssl_data * ssl_data_new(int fd, time_t timeout,
 		timeout_value = mailstream_network_delay.tv_sec * 1000 + mailstream_network_delay.tv_usec / 1000;
   }
   else {
-		timeout_value = timeout;
+		timeout_value = timeout * 1000;
   }
 #if GNUTLS_VERSION_NUMBER >= 0x030100
 	gnutls_handshake_set_timeout(session, timeout_value);

Vendor/libetpan/src/low-level/imap/mailimap.c

@@ -749,7 +749,13 @@ int mailimap_logout(mailimap * session)
   }
 
   r = mailimap_parse_response(session, &response);
-  if (r != MAILIMAP_NO_ERROR) {
+  if (r == MAILIMAP_ERROR_STREAM) {
+    // the response is expected to be MAILIMAP_ERROR_STREAM
+    // because the server responds with BYE so the stream
+    // is immediately closed
+    res = MAILIMAP_NO_ERROR;
+    goto close;
+  } else if (r != MAILIMAP_NO_ERROR) {
     res = r;
     goto close;
   }
@@ -2422,6 +2428,13 @@ int mailimap_starttls(mailimap * session)
 
   mailimap_response_free(response);
 
+  // Detect if the server send extra data after the STARTTLS response.
+  // This *may* be a "response injection attack".
+  if (session->imap_stream->read_buffer_len != 0) {
+      // Since it is also an IMAP protocol violation, exit.
+      return MAILIMAP_ERROR_STARTTLS;
+  }
+
   switch (error_code) {
   case MAILIMAP_RESP_COND_STATE_OK:
     return MAILIMAP_NO_ERROR;

Vendor/libetpan/src/low-level/imap/mailimap_types.c

@@ -1389,9 +1389,11 @@ void
 mailimap_mailbox_data_status_free(struct mailimap_mailbox_data_status * info)
 {
   mailimap_mailbox_free(info->st_mailbox);
-  clist_foreach(info->st_info_list, (clist_func) mailimap_status_info_free,
-		 NULL);
-  clist_free(info->st_info_list);
+  if (info->st_info_list != NULL) {
+    clist_foreach(info->st_info_list, (clist_func) mailimap_status_info_free,
+      NULL);
+    clist_free(info->st_info_list);
+  }
   free(info);
 }
 

Vendor/libetpan/src/low-level/imap/quota_parser.c

@@ -103,7 +103,7 @@ mailimap_quota_quota_resource_parse(mailstream * fd, MMAPString * buffer, struct
 }
 
 static int
-mailimap_quota_quota_list_nonempty_parse(mailstream * fd, MMAPString * buffer,
+mailimap_quota_quota_list_nonempty_parse(mailstream * fd, MMAPString * buffer, struct mailimap_parser_context * parser_ctx,
     size_t * indx, clist ** result,
     size_t progr_rate, progress_function * progr_fun)
 {
@@ -114,13 +114,13 @@ mailimap_quota_quota_list_nonempty_parse(mailstream * fd, MMAPString * buffer,
 
   cur_token = * indx;
 
-  r = mailimap_oparenth_parse(fd, buffer, NULL, &cur_token);
+  r = mailimap_oparenth_parse(fd, buffer, parser_ctx, &cur_token);
   if (r != MAILIMAP_NO_ERROR) {
     res = r;
     goto err;
   }
 
-  r = mailimap_struct_spaced_list_parse(fd, buffer, NULL,
+  r = mailimap_struct_spaced_list_parse(fd, buffer, parser_ctx,
       &cur_token, &quota_resource_list,
       &mailimap_quota_quota_resource_parse,
       (mailimap_struct_destructor *)
@@ -131,7 +131,7 @@ mailimap_quota_quota_list_nonempty_parse(mailstream * fd, MMAPString * buffer,
     goto err;
   }
 
-  r = mailimap_cparenth_parse(fd, buffer, NULL, &cur_token);
+  r = mailimap_cparenth_parse(fd, buffer, parser_ctx, &cur_token);
   if (r != MAILIMAP_NO_ERROR) {
     res = r;
     goto quota_list_free;
@@ -151,7 +151,7 @@ mailimap_quota_quota_list_nonempty_parse(mailstream * fd, MMAPString * buffer,
 }
 
 static int
-mailimap_quota_quota_list_empty_parse(mailstream * fd, MMAPString * buffer,
+mailimap_quota_quota_list_empty_parse(mailstream * fd, MMAPString * buffer, struct mailimap_parser_context * parser_ctx,
     size_t * indx, clist ** result,
     size_t progr_rate, progress_function * progr_fun)
 {
@@ -161,12 +161,12 @@ mailimap_quota_quota_list_empty_parse(mailstream * fd, MMAPString * buffer,
 
   cur_token = * indx;
 
-  r = mailimap_oparenth_parse(fd, buffer, NULL, &cur_token);
+  r = mailimap_oparenth_parse(fd, buffer, parser_ctx, &cur_token);
   if (r != MAILIMAP_NO_ERROR) {
     return r;
   }
 
-  r = mailimap_cparenth_parse(fd, buffer, NULL, &cur_token);
+  r = mailimap_cparenth_parse(fd, buffer, parser_ctx, &cur_token);
   if (r != MAILIMAP_NO_ERROR) {
     return r;
   }
@@ -183,24 +183,24 @@ mailimap_quota_quota_list_empty_parse(mailstream * fd, MMAPString * buffer,
 }
 
 static int
-mailimap_quota_quota_list_parse(mailstream * fd, MMAPString * buffer,
+mailimap_quota_quota_list_parse(mailstream * fd, MMAPString * buffer, struct mailimap_parser_context * parser_ctx,
     size_t * indx, clist ** result,
     size_t progr_rate, progress_function * progr_fun)
 {
   int r;
 
-  r = mailimap_quota_quota_list_empty_parse(fd, buffer, indx, result,
+  r = mailimap_quota_quota_list_empty_parse(fd, buffer, parser_ctx, indx, result,
       progr_rate, progr_fun);
   if (r == MAILIMAP_NO_ERROR) {
     return r;
   }
 
-  return mailimap_quota_quota_list_nonempty_parse(fd, buffer, indx, result,
+  return mailimap_quota_quota_list_nonempty_parse(fd, buffer, parser_ctx, indx, result,
       progr_rate, progr_fun);
 }
 
 static int
-mailimap_quota_quota_response_parse(mailstream * fd, MMAPString * buffer,
+mailimap_quota_quota_response_parse(mailstream * fd, MMAPString * buffer, struct mailimap_parser_context * parser_ctx,
     size_t * indx, struct mailimap_quota_quota_data ** result,
     size_t progr_rate, progress_function * progr_fun)
 {
@@ -226,7 +226,7 @@ mailimap_quota_quota_response_parse(mailstream * fd, MMAPString * buffer,
     goto err;
   }
 
-  r = mailimap_astring_parse(fd, buffer, NULL, &cur_token, &quotaroot,
+  r = mailimap_astring_parse(fd, buffer, parser_ctx, &cur_token, &quotaroot,
           progr_rate, progr_fun);
   if (r != MAILIMAP_NO_ERROR) {
     res = r;
@@ -239,7 +239,7 @@ mailimap_quota_quota_response_parse(mailstream * fd, MMAPString * buffer,
     goto quotaroot_free;
   }
 
-  r = mailimap_quota_quota_list_parse(fd, buffer, &cur_token,
+  r = mailimap_quota_quota_list_parse(fd, buffer, parser_ctx, &cur_token,
       &quota_list, progr_rate, progr_fun);
   if (r != MAILIMAP_NO_ERROR) {
     res = r;
@@ -268,7 +268,7 @@ mailimap_quota_quota_response_parse(mailstream * fd, MMAPString * buffer,
 }
 
 static int
-mailimap_quota_quotaroot_response_parse(mailstream * fd, MMAPString * buffer,
+mailimap_quota_quotaroot_response_parse(mailstream * fd, MMAPString * buffer, struct mailimap_parser_context * parser_ctx,
     size_t * indx, struct mailimap_quota_quotaroot_data ** result,
     size_t progr_rate, progress_function * progr_fun)
 {
@@ -295,7 +295,7 @@ mailimap_quota_quotaroot_response_parse(mailstream * fd, MMAPString * buffer,
     goto err;
   }
 
-  r = mailimap_mailbox_parse(fd, buffer, NULL, &cur_token, &mailbox,
+  r = mailimap_mailbox_parse(fd, buffer, parser_ctx, &cur_token, &mailbox,
           progr_rate, progr_fun);
   if (r != MAILIMAP_NO_ERROR) {
     res = r;
@@ -317,7 +317,7 @@ mailimap_quota_quotaroot_response_parse(mailstream * fd, MMAPString * buffer,
       goto quotaroot_list_free;
     }
 
-    r = mailimap_astring_parse(fd, buffer, NULL, &cur_token, &quotaroot,
+    r = mailimap_astring_parse(fd, buffer, parser_ctx, &cur_token, &quotaroot,
         progr_rate, progr_fun);
     if (r != MAILIMAP_NO_ERROR) {
       res = r;
@@ -386,15 +386,15 @@ int mailimap_quota_parse(int calling_parser, mailstream * fd,
   switch (calling_parser)
   {
     case MAILIMAP_EXTENDED_PARSER_MAILBOX_DATA:
-      r = mailimap_quota_quota_response_parse(fd, buffer, indx,
+      r = mailimap_quota_quota_response_parse(fd, buffer, parser_ctx, indx,
         &quota_data, progr_rate, progr_fun);
       if (r == MAILIMAP_NO_ERROR) {
 	type = MAILIMAP_QUOTA_TYPE_QUOTA_DATA;
 	data = quota_data;
       }
 
       if (r == MAILIMAP_ERROR_PARSE) {
-	r = mailimap_quota_quotaroot_response_parse(fd, buffer, indx,
+	r = mailimap_quota_quotaroot_response_parse(fd, buffer, parser_ctx, indx,
           &quotaroot_data, progr_rate, progr_fun);
         if (r == MAILIMAP_NO_ERROR) {
           type = MAILIMAP_QUOTA_TYPE_QUOTAROOT_DATA;

Vendor/libetpan/src/low-level/mime/mailmime_write_generic.c

@@ -81,6 +81,9 @@ static int mailmime_version_write_driver(int (* do_write)(void *, const char *,
 static int mailmime_encoding_write_driver(int (* do_write)(void *, const char *, size_t), void * data, int * col,
 				   struct mailmime_mechanism * encoding);
 
+static int mailmime_location_write_driver(int (* do_write)(void *, const char *, size_t), void *data, int *col,
+                                   char *location);
+
 static int mailmime_language_write_driver(int (* do_write)(void *, const char *, size_t), void * data, int * col,
 				   struct mailmime_language * language);
 
@@ -169,6 +172,10 @@ static int mailmime_field_write_driver(int (* do_write)(void *, const char *, si
     r = mailmime_language_write_driver(do_write, data, col, field->fld_data.fld_language);
     break;
 
+  case MAILMIME_FIELD_LOCATION:
+    r = mailmime_location_write_driver(do_write, data, col, field->fld_data.fld_location);
+    break;
+
   default:
     r = MAILIMF_ERROR_INVAL;
     break;
@@ -310,6 +317,33 @@ static int mailmime_encoding_write_driver(int (* do_write)(void *, const char *,
   return MAILIMF_NO_ERROR;
 }
 
+static int mailmime_location_write_driver(int (* do_write)(void *, const char *, size_t), void *data, int *col,
+                                   char *location)
+{
+  int r;
+  int len = strlen(location);
+
+  r = mailimf_string_write_driver(do_write, data, col, "Content-Location: ", 18);
+  if (r != MAILIMF_NO_ERROR)
+    return r;
+
+  if (*col > 1 && *col + len > MAX_MAIL_COL) {
+    r = mailimf_string_write_driver(do_write, data, col, "\r\n ", 3);
+    if (r != MAILIMF_NO_ERROR)
+      return r;
+  }
+
+  r = mailimf_string_write_driver(do_write, data, col, location, len);
+  if (r != MAILIMF_NO_ERROR)
+    return r;
+
+  r = mailimf_string_write_driver(do_write, data, col, "\r\n", 2);
+  if (r != MAILIMF_NO_ERROR)
+    return r;
+
+  return MAILIMF_NO_ERROR;
+}
+
 static int mailmime_language_write_driver(int (* do_write)(void *, const char *, size_t), void * data, int * col,
 				   struct mailmime_language * language)
 {

Vendor/libetpan/src/low-level/pop3/mailpop3.c

@@ -959,6 +959,14 @@ int mailpop3_stls(mailpop3 * f)
 
   if (r != RESPONSE_OK)
     return MAILPOP3_ERROR_STLS_NOT_SUPPORTED;
+
+  // Detect if the server send extra data after the STLS response.
+  // This *may* be a "response injection attack".
+  if (f->pop3_stream->read_buffer_len != 0) {
+    // Since it is also protocol violation, exit.
+    // There is no error type for STARTTLS errors in POP3
+    return MAILPOP3_ERROR_SSL;
+  }
 
   return MAILPOP3_NO_ERROR;
 }

Vendor/libetpan/src/low-level/smtp/mailsmtp.c

@@ -1121,6 +1121,14 @@ int mailesmtp_starttls(mailsmtp * session)
     return MAILSMTP_ERROR_STREAM;
   r = read_response(session);
 
+  // Detect if the server send extra data after the STARTTLS response.
+  // This *may* be a "response injection attack".
+  if (session->stream->read_buffer_len != 0) {
+    // Since it is also protocol violation, exit.
+    // There is no general error type for STARTTLS errors in SMTP
+    return MAILSMTP_ERROR_SSL;
+  }
+
   switch (r) {
   case 220:
     return MAILSMTP_NO_ERROR;

Vendor/libetpan/tests/frm-simple.c

@@ -59,7 +59,7 @@ static void simple_print_mail_info(mailmessage * msg)
 
   strip_crlf(dsp_subject);
 
-  snprintf(output, MAX_OUTPUT, "%3i: %-21.21s %-53.53s\n",
+  snprintf(output, MAX_OUTPUT, "%3i: %-21.21s %-52.52s\n",
       msg->msg_index % 1000, dsp_from, dsp_subject);
 
   printf("%s\n", output);

Vendor/mailcore2/src/core/abstract/MCMessageConstants.h

@@ -70,13 +70,14 @@ namespace mailcore {
         IMAPMessagesRequestKindHeaders       = 1 << 1,
         IMAPMessagesRequestKindStructure     = 1 << 2,
         IMAPMessagesRequestKindInternalDate  = 1 << 3,
-        IMAPMessagesRequestKindFullHeaders   = 1 << 4,
+        IMAPMessagesRequestKindFullHeaders   = 1 << 4, //Fetches the non-parsed list of a limited set of headers.
         IMAPMessagesRequestKindHeaderSubject = 1 << 5,
         IMAPMessagesRequestKindGmailLabels   = 1 << 6,
         IMAPMessagesRequestKindGmailMessageID = 1 << 7,
         IMAPMessagesRequestKindGmailThreadID  = 1 << 8,
         IMAPMessagesRequestKindExtraHeaders  = 1 << 9,
         IMAPMessagesRequestKindSize          = 1 << 10,
+        IMAPMessagesRequestKindAllHeaders    = 1 << 11, // Unlike Full headers this will fetch all the non-parsed headers
     };
 
     enum IMAPFetchRequestType {

Vendor/mailcore2/src/core/imap/MCIMAPSession.cpp

@@ -672,18 +672,24 @@ void IMAPSession::connect(ErrorCode * pError)
             * pError = ErrorTLSNotAvailable;
             goto close;
         }
+        if (!checkCertificate()) {
+            MCLog("StartTLS ssl connect certificate ERROR %d", r);
+            * pError = ErrorCertificate;
+            goto close;
+        }
+
         break;
 
         case ConnectionTypeTLS:
         r = mailimap_ssl_connect_voip(mImap, MCUTF8(mHostname), mPort, isVoIPEnabled());
-        MCLog("ssl connect %s %u %u", MCUTF8(mHostname), mPort, r);
+        MCLog("TLS ssl connect %s %u %u", MCUTF8(mHostname), mPort, r);
         if (hasError(r)) {
             MCLog("connect error %i", r);
             * pError = ErrorConnection;
             goto close;
         }
         if (!checkCertificate()) {
-            MCLog("ssl connect certificate ERROR %d", r);
+            MCLog("TLS ssl connect certificate ERROR %d", r);
             * pError = ErrorCertificate;
             goto close;
         }
@@ -2611,7 +2617,17 @@ IMAPSyncResult * IMAPSession::fetchMessages(String * folder, IMAPMessagesRequest
         struct mailimap_section * section;
 
         imap_hdrlist = mailimap_header_list_new(hdrlist);
-        section = mailimap_section_new_header_fields(imap_hdrlist);
+        
+        /*
+         * If the IMAPRequestKindFullHeaders is set then we have to fetch the full headers 
+         * otherwise only the specified list of headers 
+         */
+        if ((requestKind & IMAPMessagesRequestKindAllHeaders) != 0) {
+           section = mailimap_section_new_header();
+        } else {
+           section = mailimap_section_new_header_fields(imap_hdrlist);
+        }
+
         fetch_att = mailimap_fetch_att_new_body_peek_section(section);
         mailimap_fetch_type_new_fetch_att_list_add(fetch_type, fetch_att);
         needsHeader = true;