crash appending list and namedtuple

@loewis

BPO	8847
Nosy	@loewis, @birkenfeld, @rhettinger, @terryjreedy, @jcea, @atsuoishimoto, @amauryfa, @ncoghlan, @pitrou, @vstinner, @jackdied, @tjguk, @merwok, @alex, @bitdancer, @briancurtin, @skrah, @meadori
Files	issue8847.diff issue8847-3.2.diff issue8847-3.3.diff

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2012-08-01.09:13:02.819>
created_at = <Date 2010-05-28.22:53:45.970>
labels = ['interpreter-core', 'deferred-blocker', 'library', 'OS-windows', 'type-crash']
title = 'crash appending list and namedtuple'
updated_at = <Date 2012-08-01.13:32:32.136>
user = 'https://bugs.python.org/benrg'

bugs.python.org fields:

activity = <Date 2012-08-01.13:32:32.136>
actor = 'loewis'
assignee = 'none'
closed = True
closed_date = <Date 2012-08-01.09:13:02.819>
closer = 'loewis'
components = ['Interpreter Core', 'Library (Lib)', 'Windows']
creation = <Date 2010-05-28.22:53:45.970>
creator = 'benrg'
dependencies = []
files = ['26623', '26633', '26634']
hgrepos = []
issue_num = 8847
keywords = ['patch']
message_count = 42.0
messages = ['106695', '106720', '107103', '107108', '107111', '107112', '107115', '107116', '107117', '107119', '107120', '107124', '107147', '130233', '166907', '166908', '166909', '166929', '166935', '166936', '166942', '166943', '166946', '166954', '166964', '166969', '166971', '167009', '167026', '167033', '167036', '167037', '167057', '167062', '167063', '167098', '167100', '167101', '167106', '167111', '167120', '167127']
nosy_count = 21.0
nosy_names = ['loewis', 'georg.brandl', 'rhettinger', 'terry.reedy', 'jcea', 'ishimoto', 'amaury.forgeotdarc', 'ncoghlan', 'pitrou', 'vstinner', 'jackdied', 'tim.golden', 'eric.araujo', 'mrabarnett', 'alex', 'r.david.murray', 'brian.curtin', 'skrah', 'meador.inge', 'benrg', 'python-dev']
pr_nums = []
priority = 'deferred blocker'
resolution = 'fixed'
stage = None
status = 'closed'
superseder = None
type = 'crash'
url = 'https://bugs.python.org/issue8847'
versions = ['Python 3.2', 'Python 3.3']

c:\>python
  Python 3.1.2 (r312:79149, Mar 21 2010, 00:41:52) [MSC v.1500 32 bit (Intel)] on win32
  Type "help", "copyright", "credits" or "license" for more information.
  >>> from collections import namedtuple
  >>> foo = namedtuple('foo', '')
  >>> [1] + foo()

At this point the interpreter crashes. Also happens when foo has named arguments, and in batch scripts. foo() + [1] throws a TypeError as expected. [] + foo() returns (). The immediate cause of the crash is the CALL instruction at 1E031D5A in python31.dll jumping into uninitialized memory.

I can't reproduce this on either 3.1.2 or py3k trunk.

Running the exact same binary on winxp with an amd athlon processor,
I *did* get a crash after about 5 seconds. "python.exe has encountered a problem and needs to close. We are sorry for the inconvenience."
Trying again with IDLE instead of the command window, I get the same message (for pythonw instead of python) and the shell restarts when I close the message.

Even though foo()+[1] correctly raises a TypeError, the reverse [1] + foo() very bizarrely produces a length 1 tuple whose bizarre member is supposedly an instance of a empty list [] with length 1

from collections import namedtuple
foo = namedtuple('foo', '')
a = [1] + foo()
b=a[0]
print (type(a), len(a), type(b), len(type(b)), type(type(b)))

# <class 'tuple'> 1 [] 1 <class 'list'>

([2]+foo() produces same output)

Other than the above, any attempt to do anything with b or type(b) that I tried crashes. I presume that this is due to attribute lookups on the invalid type(b) (or something like that), which type(b) must bypass. In particular, the OP's crash is due to trying to print the tuple which tries to print its member which looks for type(b).__str__ which crashes.

To anyone: investigating crashers like this is a lot easier with IDLE and an edit window than with the interactive command window.

More experiments

from collections import namedtuple
foo = namedtuple('foo', '')
a = [] + foo()
print (a, type(a), len(a))
# () <class 'tuple'> 0

ie, a standard empty tuple, whereas
a = [1,1] + foo()
crashes immediately. So the behavior of list()+namedtuple depends on the length of the list.

There are also some funny interactions. Adding
try:
a = foo()+[]
except TypeError:
print("correct TypeError")

after the 'foo = ' line in my original 5 line example causes the final print to crash, whereas adding the same 4 lines to the 4 line example at the beginning of this message does not obviously change anything.

David, since you omitted all details, I wonder if you tested in batch mode, as I did, and hence never tried to print the malformed object, or used different OS or hardware.

I can't reproduce on 3k trunk with Ubuntu 10.04, gcc 4.4.3

namedtuples are just a subclass of tuple with only two dunder methods defined (a plain __new__ with empty __slots__). Can you provoke the same behavior with plain tuples, or a subclass of tuple that looks like one of these?

class Crasher(tuple): pass

class Crasher(tuple):
  __slots__ = ()

class Crasher(tuple):
  def __new__(cls,): return tuple.__new__(cls,)
  __slots__ = ()

Substituting
foo = tuple()
TypeError: can only concatenate list (not "tuple") to list

class Crasher(tuple): pass
foo = Crasher()
a = [1] + foo
b=a[0]
print (type(a), len(a), type(b), len(type(b)), type(type(b)))

as before, so namedtuple is not, in particular, the culprit.
Other two Crasher versions do the same. I also get a delayed pythonw error message after the print that does not cause the shell to restart. This may partly be an IDLE artifact.

Two more probes:

does it also have the same strange/crashy behavior when you subclass list and concat that to a tuple?
does dropping the optimization level down to -O help? This has "compiler quirk" written all over it. The C-code for list and tuple concat are almost identical, and both start with a type check.

"can't reproduce" does not inform as to what *did* happen with which code.

More experiments:
foo = str()
TypeError: can only concatenate list (not "str") to list

class s(str): pass
foo = s()
TypeError: Can't convert 'list' object to str implicitly

Why is it trying to do that? Of course, the interpreter can (implicitly) convert list to tuple, which must be what happens in OP example.

The subclasses of tuple and str do not gain an __radd__ method. If we add one

class s(str):
    def __radd__(s,o): print('in radd, type(o)')
foo = s()
a = [1] + foo
# prints "in radd <class 'list'>"

no implicit conversion is done.

Reversing tuple and list

class Crasher(list): pass
a = () + Crasher() # or Crasher([1])
print(a, type(a), len(a))
#[] <class 'list'> 0 # or [1] <class 'list'> 1

whereas
a = (1,) + Crasher()
crashes, so not completely symmetrical

answered before you asked (yes, similar)
(same thought) I am using PSF windows installer, however that was prepared. Martin?

if the id() of the left operand is identical to the id() of the first element in the result it would strongly support compiler skulldugerry.

class Crasher(tuple): pass
foo = Crasher()
x = [1]
a = x + foo
b=a[0]

if id(b) == id(x):
  raise Exception("It's the C compiler what did it!")

The only way I can think of this coming about is the right_op getting new'd and then .extend'ing(left_op). That extend() must be going batsh*t and inserting the left_op instead of it's contained items. The C-code for extend is more fiddly than the code for concatenation so there is more room for the compiler to generate bad code.

Good try, but for one run, the ids of foo, x, a, and b are
>>> 
15719440 15717880 15273104 12266976

Antoine Pitrou <report@bugs.python.org> wrote:
> I think we want to add those tests to the test suite as well.

What's a good place? Shall we just add one of the tests to test_tuple?

Sounds good. And another of them to test_list perhaps as well :)

Also, the only person to run the tests with the PGO build will probably
be Martin just before the releases. :)

True, but other people may still run them *after* the release.

New patches with tests for 3.2 and 3.3. For 3.2 I determined empirically
that EnableCOMDATFolding="1" (and not "0") turns on NOICF.

If anyone can confirm that this is the case or has a pointer to
the relevant vcproj docs, I'd be thrilled.

Also, the only person to run the tests with the PGO build will probably
be Martin just before the releases. :)

We could set up a buildbot slave which does PGO builds, provided somebody
volunteered an installation (including VS Pro), and somebody contributed
a build script that deviates from the regular build (perhaps including some
training on the PGI). This is a separate issue, of course.

If anyone can confirm that this is the case or has a pointer to
the relevant vcproj docs, I'd be thrilled.

http://msdn.microsoft.com/de-de/library/microsoft.visualstudio.vcprojectengine.vclinkertool.enablecomdatfolding(v=vs.90).aspx

http://msdn.microsoft.com/de-de/library/microsoft.visualstudio.vcprojectengine.optfoldingtype(v=vs.90).aspx

While the actual values for the XML schema aren't documented, I expect that
they have the numeric values that they have in C++ (i.e. optFoldingDefault=0,
optNoFolding=1, optFolding=2)

To confirm, just look up the setting in the UI.

New changeset 5a8c5631463f by Martin v. Löwis in branch '2.7':
Issue bpo-8847: Disable COMDAT folding in Windows PGO builds.
http://hg.python.org/cpython/rev/5a8c5631463f

New changeset 2638ce032151 by Martin v. Löwis in branch '3.2':
Issue bpo-8847: Disable COMDAT folding in Windows PGO builds.
http://hg.python.org/cpython/rev/2638ce032151

New changeset 029cde4e58c5 by Martin v. Löwis in branch 'default':
Issue bpo-8847: Disable COMDAT folding in Windows PGO builds.
http://hg.python.org/cpython/rev/029cde4e58c5

New changeset d3afe5d8a4da by Martin v. Löwis in branch 'default':
Issue bpo-8847: Merge with 3.2
http://hg.python.org/cpython/rev/d3afe5d8a4da

Thanks for the research and the fix!

You didn't add any test for non regression??

You didn't add any test for non regression??

Please rephrase your question: what tests did I not add?
I did add the tests that Stefan proposed.

Please rephrase your question: what tests did I not add?
I did add the tests that Stefan proposed.

Ah yes, you added new tests to Python 3.2 and 3.3, but no to Python
2.7. Why not adding these new tests to Python 2.7?

Ah yes, you added new tests to Python 3.2 and 3.3, but no to Python
2.7. Why not adding these new tests to Python 2.7?

The tests don't crash Python 2.7. So they are not useful as a test whether
the bug has been worked-around. I actually don't know how to test this
compiler
bug in Python 2.7 (except for writing specific C code that tries to trigger
the bug).

Use only safe identical code folding with BOLT

Navigation Menu

Repository navigation

crash appending list and namedtuple #53093

21 remaining items

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

crash appending list and namedtuple #53093

Description

Activity

benrg commented on May 28, 2010

bitdancer commented on May 29, 2010

terryjreedy commented on Jun 4, 2010

terryjreedy commented on Jun 4, 2010

jackdied commented on Jun 4, 2010

terryjreedy commented on Jun 4, 2010

jackdied commented on Jun 5, 2010

terryjreedy commented on Jun 5, 2010

terryjreedy commented on Jun 5, 2010

jackdied commented on Jun 5, 2010

terryjreedy commented on Jun 5, 2010

21 remaining items

pitrou commented on Jul 31, 2012

skrah commented on Jul 31, 2012

loewis commented on Jul 31, 2012

loewis commented on Jul 31, 2012

python-dev commented on Aug 1, 2012

python-dev commented on Aug 1, 2012

loewis commented on Aug 1, 2012

vstinner commented on Aug 1, 2012

loewis commented on Aug 1, 2012

vstinner commented on Aug 1, 2012

loewis commented on Aug 1, 2012

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions