email package content-transfer-encoding behaviour changed

@warsaw

BPO	1525919
Nosy	@warsaw, @bitdancer, @iritkatriel

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2006-07-20.14:22:21.000>
labels = ['type-bug', 'expert-email', '3.10', '3.11', '3.9', 'library', 'docs']
title = 'email package content-transfer-encoding behaviour changed'
updated_at = <Date 2021-12-13.09:55:31.268>
user = 'https://bugs.python.org/ThomasAH'

bugs.python.org fields:

activity = <Date 2021-12-13.09:55:31.268>
actor = 'iritkatriel'
assignee = 'docs@python'
closed = False
closed_date = None
closer = None
components = ['Documentation', 'Library (Lib)', 'email']
creation = <Date 2006-07-20.14:22:21.000>
creator = 'ThomasAH'
dependencies = []
files = []
hgrepos = []
issue_num = 1525919
keywords = []
message_count = 11.0
messages = ['29229', '29230', '58248', '73949', '105045', '184663', '184685', '184692', '184698', '408387', '408433']
nosy_count = 9.0
nosy_names = ['barry', 'ThomasAH', 'rdemetrescu', 'paulproteus', 'r.david.murray', 'docs@python', 'python-dev', 'littleq0903', 'iritkatriel']
pr_nums = []
priority = 'normal'
resolution = None
stage = None
status = 'open'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue1525919'
versions = ['Python 3.9', 'Python 3.10', 'Python 3.11']

Linked PRs

gh-43702: Replace existing Content-Transfer-Encoding in encode_base64() and encode_quopri() #158230

from email.Message import Message
from email.Charset import Charset, QP
text = "="
msg = Message()
charset = Charset("utf-8")
charset.header_encoding = QP
charset.body_encoding = QP
msg.set_charset(charset)
msg.set_payload(text)
print msg.as_string()

Gives

MIME-Version: 1.0
Content-Type: text/plain; charset="utf-8"
Content-Transfer-Encoding: quoted-printable

=3D

With the email package from python2.4.3 and 2.4.4c0 the
last '=3D' becomes just '=', so an extra
msg.body_encode(text) is needed.

Logged In: YES
user_id=839582

One program which got hit by this is MoinMoin, see
http://moinmoin.wikiwikiweb.de/MoinMoinBugs/ResetPasswordEmailImproperlyEncoded

I am not sure if it is related, but anyway...

MIMEText behaviour has changed from python 2.4 to 2.5.

# Python 2.4

>>> from email.MIMEText import MIMEText
>>> m = MIMEText(None, 'html', 'iso-8859-1')
>>> m.set_payload('abc ' * 50)
>>> print m
From nobody Thu Dec  6 12:52:40 2007
Content-Type: text/html; charset="iso-8859-1"
MIME-Version: 1.0
Content-Transfer-Encoding: quoted-printable

abc abc abc abc abc abc abc abc abc abc abc abc abc abc abc abc abc abc abc=
abc abc abc abc abc abc abc abc abc abc abc abc abc abc abc abc abc abc ab=
c abc abc abc abc abc abc abc abc abc abc abc abc=20

# Python 2.5

>>> from email.MIMEText import MIMEText
>>> m = MIMEText(None, 'html', 'iso-8859-1')
>>> m.set_payload('abc ' * 50)
>>> print m
From nobody Thu Dec  6 14:46:07 2007
Content-Type: text/html; charset="iso-8859-1"
MIME-Version: 1.0
Content-Transfer-Encoding: quoted-printable

abc abc abc abc abc abc abc abc abc abc abc abc abc abc abc abc abc abc
abc abc abc abc abc abc abc abc abc abc abc abc abc abc abc abc abc abc
abc abc abc abc abc abc abc abc abc abc abc abc abc abc

However, if we initialize MIMEText with the text, we get the correct output:

# python 2.5

>>> from email.MIMEText import MIMEText
>>> m = MIMEText('abc ' * 50, 'html', 'iso-8859-1')
>>> print m
From nobody Thu Dec  6 13:01:17 2007
Content-Type: text/html; charset="iso-8859-1"
MIME-Version: 1.0
Content-Transfer-Encoding: quoted-printable

abc abc abc abc abc abc abc abc abc abc abc abc abc abc abc abc abc abc abc=
abc abc abc abc abc abc abc abc abc abc abc abc abc abc abc abc abc abc ab=
c abc abc abc abc abc abc abc abc abc abc abc abc=20

If I want to set payload after MIMEText is already created, I need to
use this workaround::

#python 2.5
from email.MIMEText import MIMEText
m = MIMEText(None, 'html', 'iso-8859-1')
m.set_payload(m._charset.body_encode('abc' * 50))

PS: The issue's versions field is filled with "Python 2.4". Shouldn't it
be "Python 2.5" ?

Another way to see this issue is that the email module double-encodes
when one attempts to use quoted-printable encoding. This has to be
worked around by e.g. MoinMoin.

It's easy to get proper base64-encoded output of email.mime.text:

 	>>> mt = email.mime.text.MIMEText('Ta mère', 'plain', 'utf-8')
 	>>> 'Content-Transfer-Encoding: base64' in mt.as_string()
 	True
 	>>> mt.as_string().split('\n')[-2]
 	'VGEgbcOocmU='

There we go, all nice and base64'd.

I can *not* figure out how to get quoted-printable-encoding. I found
http://docs.python.org/lib/module-email.encoders.html , so I thought
great - I'll just encode my MIMEText object:

 	>>> email.encoders.encode_quopri(mt)
 	>>> 'Content-Transfer-Encoding: quoted-printable' in mt.as_string()
 	True

Great! Except it's actually double-encoded, and the headers admit to as
much. You see here that, in addition to the quoted-printable header
just discovered, there is also a base64-related header, and the result
is not strictly QP encoding but QP(base64(payload)).

 	>>> 'Content-Transfer-Encoding: base64' in mt.as_string()
 	True
 	>>> mt.as_string().split('\n')[-2]
 	'VGEgbcOocmU=3D'

It should look like:

 	>>> quopri.encodestring('Ta mère')
 	'Ta m=C3=A8re'

I raised this issue on the Baypiggies list
<http://mail.python.org/pipermail/baypiggies/2008-September/003983.html\>, but
luckily I found this here bug. This is with Python 2.5.2-0ubuntu1 from
Ubuntu 8.04.

paulproteus@alchemy:~ $ python --version
Python 2.5.2

If we can come to a decision as to how this *should* work, I could
contribute a patch and/or tests to fix it. I could even perhaps write a
new section of the Python documentation of the email module explaining this.

Roger Demetrescu, I filed the issue with "Python 2.4", because the behavior changed somewhere between 2.4.2 and 2.4.3

The updated link to the MoinMoin bug entry is:
http://moinmo.in/MoinMoinBugs/ResetPasswordEmailImproperlyEncoded

The workaround I use to be compatible with <= 2.4.2 and >= 2.4.3 is:

    msg.set_payload('=')
    if msg.as_string().endswith('='):
        text = charset.body_encode(text)
    msg.set_payload(text)

New changeset ba500b179c3a by R David Murray in branch '3.2':
bpo-1525919: Document MIMEText+set_payload encoding behavior.
http://hg.python.org/cpython/rev/ba500b179c3a

New changeset fcbc28ef96a3 by R David Murray in branch '3.3':
Merge: bpo-1525919: Document MIMEText+set_payload encoding behavior.
http://hg.python.org/cpython/rev/fcbc28ef96a3

New changeset b9e07f20832e by R David Murray in branch 'default':
Merge: bpo-1525919: Document MIMEText+set_payload encoding behavior.
http://hg.python.org/cpython/rev/b9e07f20832e

I've committed the doc change. I'm going to be lazy and leave this issue open to deal with the encodings module fix.

The encoding functions are now doing

orig = msg.get_payload(decode=True)

Does this fix the double-encoding issue?

This change was made in
00ae435

Default python3 on Debian buster:
$ python3
Python 3.7.3 (default, Jan 22 2021, 20:04:44) 
[GCC 8.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import email.mime.text
>>> mt = email.mime.text.MIMEText('Ta mère', 'plain', 'utf-8')
>>> print(mt.as_string())
Content-Type: text/plain; charset="utf-8"
MIME-Version: 1.0
Content-Transfer-Encoding: base64

VGEgbcOocmU=

>>> email.encoders.encode_quopri(mt)
>>> print(mt.as_string())
Content-Type: text/plain; charset="utf-8"
MIME-Version: 1.0
Content-Transfer-Encoding: base64
Content-Transfer-Encoding: quoted-printable

Ta=20m=C3=A8re

So the encoded text looks good now, but there are still duplicate headers.

Old output (python2.7) is identical to what Asheesh Laroia (paulproteus)
reported for python2.5:
---
Content-Type: text/plain; charset="utf-8"
MIME-Version: 1.0
Content-Transfer-Encoding: base64
Content-Transfer-Encoding: quoted-printable

VGEgbcOocmU=3D

One piece of this is still broken at main: the duplicate-header half. email.encoders.encode_base64/encode_quopri/encode_7or8bit all append Content-Transfer-Encoding instead of replacing it, so calling any of them on a message that already carries one — MIMEText sets it by default — produces two Content-Transfer-Encoding: headers in the output. The get_payload(decode=True) change linked above did fix the payload half (that landed back in bpo-18324), but I still get two CTE headers at 3.16 on a MIMEText payload, so the duplicate header is what's left. The fix is one line per encoder — del msg['Content-Transfer-Encoding'] before the assignment (del on a missing header is a no-op, so the normal path is unchanged). I can send the patch with tests.

Reproducer at current main (6df4993)

"""gh-43702: the three email.encoders append a second Content-Transfer-Encoding
header when the message already carries one (MIMEText sets one by default)."""
import email.mime.text, email.encoders

mt = email.mime.text.MIMEText('héllo wörld', 'plain', 'utf-8')
email.encoders.encode_quopri(mt)
ctes = [h for h in mt.keys() if h == 'Content-Transfer-Encoding']
print('encode_quopri CTE count:', len(ctes))        # -> 2 (base64 + quoted-printable)
print('payload ok (not double-enc):', mt.get_payload())  # -> placeholder

for enc in ('encode_base64', 'encode_7or8bit'):
    m = email.mime.text.MIMEText('hi ' * 3, 'plain', 'utf-8')
    getattr(email.encoders, enc)(m)
    print(enc, 'CTE count:', len([h for h in m.keys() if h == 'Content-Transfer-Encoding']))  # -> 2

# Fix: `del msg['Content-Transfer-Encoding']` before assigning (no-op when absent).
mt = email.mime.text.MIMEText('héllo wörld', 'plain', 'utf-8')
orig = mt.get_payload(decode=True)
mt.set_payload(email.encoders._qencode(orig))
del mt['Content-Transfer-Encoding']
mt['Content-Transfer-Encoding'] = 'quoted-printable'
print('fixed CTE count:', len([h for h in mt.keys() if h == 'Content-Transfer-Encoding']))  # -> 1

encode_quopri CTE count: 2
payload ok (not double-enc): h=C3=A9llo=20w=C3=B6rld
encode_base64 CTE count: 2
encode_7or8bit CTE count: 2
fixed CTE count: 1

Navigation Menu

Repository navigation

email package content-transfer-encoding behaviour changed #43702

Linked PRs

18 remaining items

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

email package content-transfer-encoding behaviour changed #43702

Description

Linked PRs

Activity

ThomasAH commented on Jul 20, 2006

ThomasAH commented on Jul 20, 2006

rdemetrescu commented on Dec 6, 2007

paulproteus commented on Sep 27, 2008

ThomasAH commented on May 5, 2010

18 remaining items

python-dev commented on Mar 19, 2013

bitdancer commented on Mar 19, 2013

iritkatriel commented on Dec 12, 2021

ThomasAH commented on Dec 13, 2021

soreavis commented on Jul 19, 2026

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions