email: a header value that is itself a well-formed RFC 2047 encoded word is decoded or written verbatim, never encoded #160148

Bug description

With the modern (email.policy.default) API there is no way to send a Subject whose literal text is =?utf-8?q?x?=. Setting it on a message decodes it on the spot, and folding it as a string writes it out unchanged, where any RFC 2047 reader will decode it:

>>> import email.message, email.policy
>>> m = email.message.EmailMessage()
>>> m['Subject'] = '=?utf-8?q?x?='
>>> m['Subject']
'x'
>>> m.as_string()
'Subject: x\n\n'
>>> email.policy.default.fold('Subject', '=?utf-8?q?x?=')
'Subject: =?utf-8?q?x?=\n'   # a reader decodes this to 'x' as well
>>> email.policy.default.fold('Subject', '=?utf-8?q?x?=\xe9')
'Subject: x=?utf-8?q?=C3=A9?=\n'

The legacy API gets this right:

>>> import email.header
>>> email.header.Header('=?utf-8?q?x?=', 'utf-8').encode()
'=?utf-8?b?PT91dGYtOD9xP3g/PQ==?='

RFC 2047 §7 says that a composer "MUST ensure that any string of non-white-space printable ASCII characters within a *text or *ctext that begins with =? and ends with ?= be a valid encoded-word", and §5 anticipates the case: "In rare cases it may be necessary to encode ordinary text that looks like an encoded-word." The only compliant output for the literal text above is an encoded word whose decoded value is =?utf-8?q?x?=, as the legacy API produces.

Apparently both behaviors follow the current documentation, so I guess this needs some discussion. UnstructuredHeader is documented to decode encoded words in whatever string it is given, which is what happens in __setitem__ via header_factory (get_unstructured treats every =? as a possible encoded word). And EmailPolicy.fold is documented to treat a plain string as a "source value", so with refold_source='long' a short value is emitted unchanged, and when it is refolded it goes through the same parser (policy.py), which is where the x=?utf-8?q?=C3=A9?= output comes from. The gap is that the API has no way to say "this string is the decoded text, not wire source", so any text supplied to the program is interpreted as the wire source.

Possible directions, none of which I have a strong opinion on:

• Treat strings set by a program as decoded text (and encode look-alikes on output, per RFC 2047 §7). This would change the documented decoding on __setitem__, which some code may rely on to round-trip raw header text.
• Keep the current behavior and document the limitation, possibly with an escape hatch (a header object constructed from decoded text).

This came out of the Hypothesis round-trip test in #159771, which originally carried it as an expectedFailure; it was taken out of that PR at the reviewer's request so that the question can be discussed separately. The test, for reference (a fold round trip, written against test_policy.py):

@hypothesis.given(
    word=hypothesis.strategies.text(alphabet='abc', min_size=1, max_size=5),
    trailer=hypothesis.strategies.sampled_from(['', ')', ' word']),
)
@hypothesis.example(word='x', trailer='')
@hypothesis.example(word='x', trailer=')')
def test_fold_encoded_word_lookalike_round_trip(self, word, trailer):
    subject = f'=?utf-8?q?{word}?={trailer}'
    folded = email.policy.default.fold('Subject', subject)
    reparsed = email.message_from_string(
        folded, policy=email.policy.default)['Subject']
    self.assertEqual(reparsed, subject)

Possibly related: #85725 (an encoded word produced by folding that violates RFC 2047), though that is about the generator's output rather than user text.

CPython versions tested on

CPython main (6e1bd24)

Operating systems tested on

Linux