Skip to content

gh-107862: Fix infinite loop in email header folding found by a Hypothesis round-trip test - #159771

Open
pganssle wants to merge 2 commits into
python:mainfrom
pganssle:email-fold-hypothesis
Open

pganssle wants to merge 2 commits into
python:mainfrom
pganssle:email-fold-hypothesis

Conversation

@pganssle

@pganssle pganssle commented Oct 19, 2026 •

Copy link
Copy Markdown
Member

Stacked on #159770; only the top two commits are new here.

When an encoded word does not fit in the remaining space of a line, _fold_as_ew shrinks the chunk being encoded one character at a time until it fits. Nothing stops it from shrinking the chunk to nothing, and an empty chunk makes no progress: the loop appends an empty encoded word and a new continuation line forever.

>>> import email.policy
>>> email.policy.default.clone(max_line_length=20).fold('Subject', '😀')
# never returns

This is reachable whenever a single character's encoded form is wider than a fresh continuation line, e.g. any character outside the BMP at max_line_length=20 (the q-encoded word is 24 characters). The two earlier fixes in this area (bpo-33529, bpo-36564) handled lines too short for the RFC 2047 chrome, but not lines too short for the chrome plus one character.

The fix keeps at least one character in the chunk, so the loop always advances and the line runs over max_line_length instead. That matches what _refold_parse_tree already does when it cannot fold a part at all, and avoids raising for widths that work for every other input.

Tests, next to the two earlier infinite-loop regressions in test_policy.py:

  • An exact-output regression case, as the earlier two have.
  • A Hypothesis property: folding any generated unstructured value at any width in 1..120 (plus 0 and None) either raises HeaderParseError (only possible at widths of 13 or fewer, where no encoded word fits) or reparses to the same value modulo leading whitespace, which the parser drops. 100 examples take about 0.2 s; the explicit examples replay under the stubs.
  • A separate expectedFailure test for a second finding: a header value that is itself a well-formed encoded word is written verbatim, so reparsing decodes =?utf-8?q?x?= to x. As far as I can tell RFC 2047 §5 requires such text to be encoded. Recorded rather than worked around, so it starts failing when fixed; a separate issue for it is coming.

@bedevere-app bedevere-app Bot added the type-feature A feature request or enhancement label Oct 19, 2026
Comment thread Lib/test/support/_hypothesis_stubs/stateful.py
Comment thread Lib/test/support/_hypothesis_stubs/stateful.py Outdated
Comment thread Lib/test/support/hypothesis_helper.py Outdated
Comment thread Lib/test/test_hypothesis_stubs.py Outdated
@pganssle
pganssle force-pushed the email-fold-hypothesis branch from ad9482f to aeb677f Compare October 19, 2026 21:30
…he line

When an encoded word does not fit in the remaining space of a line,
_fold_as_ew shrinks the chunk being encoded one character at a time
until it fits. Nothing stops it from shrinking the chunk to nothing,
and an empty chunk makes no progress: the loop appends an empty
encoded word and a new continuation line forever.

This is reachable whenever a single character's encoded form is wider
than a fresh continuation line, e.g. any character outside the BMP
with max_line_length=20 (the q-encoded word is 24 characters). The two
earlier fixes in this area (bpo-33529, bpo-36564) handled lines too
short for the RFC 2047 chrome, but not lines too short for the chrome
plus one character.

This keeps at least one character in the chunk, so the loop always
advances and the line runs over max_line_length instead. That matches
what _refold_parse_tree already does when it cannot fold a part at
all, and avoids raising for widths that work for every other input.

The regression test pins the exact output, as the earlier two did. The
Hypothesis property checks that folding any unstructured value at any
width either raises HeaderParseError (only possible at widths of 13 or
fewer, where no encoded word fits) or reparses to the same value modulo
leading whitespace, which the parser drops.
…oded words

A header value that happens to be a well-formed RFC 2047 encoded word
is written out verbatim, so reparsing the message decodes it:
folding 'Subject' '=?utf-8?q?x?=' comes back as 'x'. As far as I can
tell RFC 2047 section 5 requires an encoder to encode such text.

This records the behavior as an expected failure rather than keeping
the round trip property away from it, so the test starts failing when
the generator is fixed.
@pganssle
pganssle force-pushed the email-fold-hypothesis branch from 7f30bad to 61e55a2 Compare October 20, 2026 18:01
@pganssle pganssle added type-bug An unexpected behavior, bug, or error and removed type-feature A feature request or enhancement labels Oct 20, 2026
@pganssle
pganssle marked this pull request as ready for review October 20, 2026 18:02
@pganssle
pganssle requested a review from a team as a code owner October 20, 2026 18:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

awaiting core review type-bug An unexpected behavior, bug, or error

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant