mail message parsing glitch

@warsaw

BPO	1590744
Nosy	@warsaw, @bitdancer, @akheron
Dependencies	bpo-14731: Enhance Policy framework in preparation for adding email6 policies as provisional
Files	mbox: mailbox preserve_leading_whitespace.patch

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2006-11-05.09:21:04.000>
labels = ['type-bug', 'expert-email']
title = 'mail message parsing glitch'
updated_at = <Date 2012-06-18.08:03:55.567>
user = 'https://bugs.python.org/mcspang'

bugs.python.org fields:

activity = <Date 2012-06-18.08:03:55.567>
actor = 'petri.lehtinen'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['email']
creation = <Date 2006-11-05.09:21:04.000>
creator = 'mcspang'
dependencies = ['14731']
files = ['2204', '21370']
hgrepos = []
issue_num = 1590744
keywords = ['patch']
message_count = 4.0
messages = ['30445', '30446', '131976', '161476']
nosy_count = 4.0
nosy_names = ['barry', 'mcspang', 'r.david.murray', 'petri.lehtinen']
pr_nums = []
priority = 'normal'
resolution = None
stage = 'test needed'
status = 'open'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue1590744'
versions = ['Python 3.3']

There's something wrong with the handling of line
continuation in headers.

In the attached mbox the 'Message-id' header is read
and stored with a leading space. So message_id = '
<9B09D75DF5B3494BA06E6FE478CE9CC10526CFAF@mgtserver3.ontario.int.ec.gc.ca>'
when it's first read.

If you write this message out to a new mbox, it is
written with the leading space and without the newline.

THEN, if you read it in AGAIN, the parser ignores the
leading space.

One of these steps is buggy, I'm not sure which. It
seems to me that the value returned by
msg['Message_id'] should not change when the file is
written then re-read. In the initial read and final
reads above the value differs by a space.

This makes some sense, although it may not be ideal. The "leading space" is really an RFC 2822 continuation line, so that whitespace at the start of the second line is folding whitespace. Technically those two lines should (and do) get collapsed.

The question is whether this message should be printed idempotently. In general the email package tries hard to maintain this principle, so I think it's probably a valid bug. However, it doesn't affect the semantics of the message so it's not a high priority fix.

I needed an airplane-trip-sized problem to work on on the way back from PyCon and the sprints, so I tried my hand at "fixing" this. The attached patch is really just a proof of concept. Because it is so invasive of the email package machinery I doubt that I will apply it, but it does serve to prove that it is quite practical, given the right design, to preserve the leading whitespace in message headers, and this does enable the email package to read and write the messages in the sample mbox without changing them. I will incorporate what I learned from this exercise into the header management in email6.

On the other hand, if anyone else thinks this *is* worth tidying up an applying I could be convinced.

Note that after this patch one test fails, but that test failure is actually a buggy test that hides a bug in the header formatter (a failure to provide folding white space at the start of a continuation line). That bug I may revisit.

I think I can actually fix this once the patch in bpo-14731 is applied.

This looks fixed at main. The reported case — a Message-id: whose value sits on a folded continuation line — now round-trips cleanly: msg['Message-id'] comes back without the leading newline/space under both compat32 and email.policy.default, and parse → serialize → parse is idempotent on both (including via the mailbox path from the original report). The strip is recent: #124452 (Nov 2024) changed Compat32.header_source_parse to join continuation lines and strip exactly the leading whitespace that caused the 2006 non-idempotency — which is why older checks still showed it broken. There's no dedicated regression test for the all-on-a-continuation-line case in TestIdempotent; I can add one if that's worth having.

Round-trip check at current main (6df4993)

import email
import email.policy
import io
import mailbox
import tempfile
import os

# Message 1 from the sample mbox: Message-id value folded onto a continuation
# line (header name, EOL, then leading-whitespace continuation).
MSG1 = (
    "To: <someone@example.com>\r\n"
    "Message-id:\r\n"
    " <9B09D75DF5B3494BA06E6FE478CE9CC10526CFAF@mgtserver3.ontario.int.ec.gc.ca>\r\n"
    "\r\n"
    "Hello.\r\n"
)


def roundtrip(policy_name, policy):
    print(f"\n===== policy: {policy_name} =====")
    msg = email.message_from_string(MSG1, policy=policy)
    v1 = msg["Message-id"]
    print("initial   msg['Message-id'] =", repr(v1))

    out = msg.as_string()
    print("serialized message:")
    for line in out.splitlines(keepends=True):
        print("   ", repr(line))

    msg2 = email.message_from_string(out, policy=policy)
    v2 = msg2["Message-id"]
    print("re-parsed msg['Message-id'] =", repr(v2))

    # str() forces modern header objects to render for comparison
    print("idempotent str(v1)==str(v2)?", str(v1) == str(v2))
    print("idempotent serialize stable?", msg2.as_string() == out)
    return out, msg2.as_string()


compat_out, compat_re = roundtrip("compat32 (default)", email.policy.compat32)
mod_out, mod_re = roundtrip("email.policy.default", email.policy.default)


# ---- Now the mailbox path the original reporter used ----
def mailbox_roundtrip(factory_policy_label):
    print(f"\n===== mailbox path ({factory_policy_label}) =====")
    d = tempfile.mkdtemp()
    p1 = os.path.join(d, "box1")
    with open(p1, "w") as f:
        f.write("From - Sat Oct 14 00:11:01 2006\n")
        f.write(MSG1.replace("\r\n", "\n"))
    box = mailbox.mbox(p1)
    m = box[0]
    v1 = m["Message-id"]
    print("mbox read msg['Message-id'] =", repr(v1))
    p2 = os.path.join(d, "box2")
    box2 = mailbox.mbox(p2)
    box2.add(m)
    box2.flush()
    with open(p2) as f:
        written = f.read()
    print("written box2:")
    for line in written.splitlines(keepends=True):
        print("   ", repr(line))
    box3 = mailbox.mbox(p2)
    v2 = box3[0]["Message-id"]
    print("mbox re-read msg['Message-id'] =", repr(v2))
    print("mailbox idempotent?", str(v1) == str(v2))


mailbox_roundtrip("default compat32")

===== policy: compat32 (default) =====
initial   msg['Message-id'] = '<9B09D75DF5B3494BA06E6FE478CE9CC10526CFAF@mgtserver3.ontario.int.ec.gc.ca>'
serialized message:
    'To: <someone@example.com>\n'
    'Message-id: <9B09D75DF5B3494BA06E6FE478CE9CC10526CFAF@mgtserver3.ontario.int.ec.gc.ca>\n'
    '\n'
    'Hello.\n'
re-parsed msg['Message-id'] = '<9B09D75DF5B3494BA06E6FE478CE9CC10526CFAF@mgtserver3.ontario.int.ec.gc.ca>'
idempotent str(v1)==str(v2)? True
idempotent serialize stable? True

===== policy: email.policy.default =====
initial   msg['Message-id'] = '<9B09D75DF5B3494BA06E6FE478CE9CC10526CFAF@mgtserver3.ontario.int.ec.gc.ca>'
serialized message:
    'To: <someone@example.com>\n'
    'Message-id:\n'
    ' <9B09D75DF5B3494BA06E6FE478CE9CC10526CFAF@mgtserver3.ontario.int.ec.gc.ca>\n'
    '\n'
    'Hello.\n'
re-parsed msg['Message-id'] = '<9B09D75DF5B3494BA06E6FE478CE9CC10526CFAF@mgtserver3.ontario.int.ec.gc.ca>'
idempotent str(v1)==str(v2)? True
idempotent serialize stable? True

===== mailbox path (default compat32) =====
mbox read msg['Message-id'] = '<9B09D75DF5B3494BA06E6FE478CE9CC10526CFAF@mgtserver3.ontario.int.ec.gc.ca>'
written box2:
    'From - Sat Oct 14 00:11:01 2006\n'
    'To: <someone@example.com>\n'
    'Message-id: <9B09D75DF5B3494BA06E6FE478CE9CC10526CFAF@mgtserver3.ontario.int.ec.gc.ca>\n'
    '\n'
    'Hello.\n'
    '\n'
mbox re-read msg['Message-id'] = '<9B09D75DF5B3494BA06E6FE478CE9CC10526CFAF@mgtserver3.ontario.int.ec.gc.ca>'
mailbox idempotent? True

Navigation Menu

Repository navigation

mail message parsing glitch #44196

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

mail message parsing glitch #44196

Description

Activity

mcspang commented on Nov 5, 2006

warsaw commented on Mar 12, 2007

bitdancer commented on Mar 24, 2011

bitdancer commented on May 24, 2012

soreavis commented on Jul 19, 2026

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions