IndexError in email parse_message_id

[tmmcguire]

Bug report

I apologize in advance for this bug report. It's unfortunately short on details.

In the email module, in file _header_value_parser.py, I am seeing a HeaderParseError: "expected atom at a start of dot-atom-text but found '[83c48dddbea7492e873224a5ae1c04be...'" that triggers an IndexError: list index out of range.

Unfortunately, this is happening in the REST server for Mailman's HyperKitty, so the only information I have is:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/email/_header_value_parser.py", line 2069, in get_msg_id
    token, value = get_dot_atom_text(value)
  File "/usr/local/lib/python3.8/email/_header_value_parser.py", line 1334, in get_dot_atom_text
    raise errors.HeaderParseError("expected atom at a start of "
email.errors.HeaderParseError: expected atom at a start of dot-atom-text but found '[83c48dddbea7492e873224a5ae1c04be-JFBVALKQOJXWILKNK4YVA7CBPJ2XEZKEMV3E64DTPRCW2YLJNR6EK6DPKNWXI4A=@microsoft.com]>'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/app/mailman/venv/lib/python3.8/site-packages/django/core/handlers/exception.py", line 34, in inner
    response = get_response(request)
  File "/app/mailman/venv/lib/python3.8/site-packages/django/core/handlers/base.py", line 115, in _get_response
    response = self.process_exception_by_middleware(e, request)
  File "/app/mailman/venv/lib/python3.8/site-packages/django/core/handlers/base.py", line 113, in _get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "/app/mailman/venv/lib/python3.8/site-packages/django/views/decorators/http.py", line 40, in inner
    return func(request, *args, **kwargs)
  File "/app/mailman/venv/lib/python3.8/site-packages/hyperkitty/views/mailman.py", line 118, in _decorator
    return func(request, *args, **kwargs)
  File "/app/mailman/venv/lib/python3.8/site-packages/django/views/decorators/csrf.py", line 54, in wrapped_view
    return view_func(*args, **kwargs)
  File "/app/mailman/venv/lib/python3.8/site-packages/hyperkitty/views/mailman.py", line 164, in archive
    add_to_list(mlist_fqdn, msg)
  File "/app/mailman/venv/lib/python3.8/site-packages/hyperkitty/lib/incoming.py", line 65, in add_to_list
    msg_id = get_message_id(message)
  File "/app/mailman/venv/lib/python3.8/site-packages/hyperkitty/lib/utils.py", line 60, in get_message_id
    msg_id = email.utils.unquote(re.sub(r'\s', '', message['Message-Id']))
  File "/usr/local/lib/python3.8/email/message.py", line 391, in __getitem__
    return self.get(name)
  File "/usr/local/lib/python3.8/email/message.py", line 471, in get
    return self.policy.header_fetch_parse(k, v)
  File "/usr/local/lib/python3.8/email/policy.py", line 163, in header_fetch_parse
    return self.header_factory(name, value)
  File "/usr/local/lib/python3.8/email/headerregistry.py", line 607, in __call__
    return self[name](name, value)
  File "/usr/local/lib/python3.8/email/headerregistry.py", line 202, in __new__
    cls.parse(value, kwds)
  File "/usr/local/lib/python3.8/email/headerregistry.py", line 535, in parse
    kwds['parse_tree'] = parse_tree = cls.value_parser(value)
  File "/usr/local/lib/python3.8/email/_header_value_parser.py", line 2126, in parse_message_id
    token, value = get_msg_id(value)
  File "/usr/local/lib/python3.8/email/_header_value_parser.py", line 2073, in get_msg_id
    token, value = get_obs_local_part(value)
  File "/usr/local/lib/python3.8/email/_header_value_parser.py", line 1516, in get_obs_local_part
    if (obs_local_part[0].token_type == 'dot' or
IndexError: list index out of range

The HeaderParseError is apparently leaving obs_local_part in a bad state (the expression following line 1516 includes obs_local_part[0] and obs_local_part[1].

I have not been able to find the email text to find out what the Message-Id header actually is.

Your environment

Python 3.8.7 (default, Nov 15 2022, 11:52:53)

[GCC 8.5.0 20210514 (Red Hat 8.5.0-15)] on linux

GNU Mailman 3.3.5 (Tom Sawyer)

Red Hat Enterprise Linux release 8.7 (Ootpa)

@warsaw , @bitdancer , @maxking (You are in the Experts Index.)

Linked PRs

• gh-105802: email._header_value_parser: prevent IndexError in get_obs_local_part #108133

[fsc-eriker]

If you scroll far enough to the right, the traceback actually reveals the (problematic part of) the message-id.

I have seen this in Microsoft-generated messages recently; they incorrectly create square brackets around the message-id.

Here is a simple test message which reproduces the problem.

From: Smooth Criminal <[email protected]>
Subject: We can haz broken message-idz
Message-id: <[83c48dddbea7492e873224a5ae1c04be-JFBVALKQOJXWILKNK4YVA7CBPJ2XEZKEMV3E64DTPRCW2YLJNR6EK6DPKNWXI4A=@microsoft.com]>

Hello from sunny^W Redmond!

You need to trigger Message-Id parsing to exhibit the problem. Try this to repro.

import email
from email.policy import default

with open("message.eml", "rb") as msgbytes:
    message = email.message_from_binary_file(msgbytes, policy=default)
msgid = message["message-id"]

with the sample message in message.eml