Built-in module _io can lose data from buffered files in reference cycles

@tim-one

BPO	17852
Nosy	@tim-one, @nascheme, @pitrou, @vstinner, @njsmith, @methane, @vadmium, @serhiy-storchaka, @jstasiak, @xgid, @nitishch, @Arusekk, @miss-islington, @Volker-Weissmann
PRs	bpo-17852: Maintain a list of BufferedWriter objects. Flush them on exit. #1908 Revert "bpo-17852: Maintain a list of BufferedWriter objects. Flush them on exit." #3337 bpo-17852: Maintain a list of BufferedWriter objects. Flush them on exit. #3372 bpo-17852: Maintain a list of BufferedWriter objects. Flush them on exit. #3372 bpo-17852: Revert incorrect fix based on misunderstanding of _Py_PyAtExit() semantics #4826 bpo-17852: Fix PR #3372, flush BufferedWriter objects on exit. #4847 bpo-17852: Fixed the documentation about closing files #23135 [3.9] bpo-17852: Doc: Fix the tutorial about closing files (GH-23135) #23527 [3.8] bpo-17852: Doc: Fix the tutorial about closing files (GH-23135) #23528
Files	y.py gcio.py pyio.diff buffer_crash.py buffer_not_flushed.py buffer_register_flush.txt

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/nascheme'
closed_at = None
created_at = <Date 2013-04-27.07:33:30.917>
labels = ['type-bug', '3.7', 'expert-IO']
title = 'Built-in module _io can lose data from buffered files in reference cycles'
updated_at = <Date 2020-12-01.10:53:50.492>
user = 'https://github.com/arigo'

bugs.python.org fields:

activity = <Date 2020-12-01.10:53:50.492>
actor = 'miss-islington'
assignee = 'nascheme'
closed = False
closed_date = None
closer = None
components = ['IO']
creation = <Date 2013-04-27.07:33:30.917>
creator = 'arigo'
dependencies = []
files = ['30030', '37357', '37359', '47121', '47332', '47340']
hgrepos = []
issue_num = 17852
keywords = ['patch']
message_count = 71.0
messages = ['187890', '187898', '187903', '187922', '187966', '188075', '188077', '188095', '188130', '188137', '231376', '231386', '231391', '231420', '232103', '232109', '232113', '232114', '232116', '232117', '232118', '232120', '232121', '232122', '232123', '232124', '232126', '232127', '232129', '232131', '232132', '232133', '232134', '232137', '232139', '232140', '232146', '294868', '294910', '294956', '301295', '301297', '301329', '301331', '301353', '301361', '301363', '302758', '305653', '305665', '308170', '308177', '308236', '308330', '308332', '308412', '308417', '308419', '308424', '308666', '308668', '308681', '308787', '310602', '310648', '310659', '355064', '381924', '381925', '381926', '382232']
nosy_count = 14.0
nosy_names = ['tim.peters', 'nascheme', 'pitrou', 'vstinner', 'njs', 'methane', 'martin.panter', 'serhiy.storchaka', 'jstasiak', 'xgdomingo', 'nitishch', 'Arusekk', 'miss-islington', 'Volker Wei\xc3\x9fmann']
pr_nums = ['1908', '3337', '3372', '3372', '4826', '4847', '23135', '23527', '23528']
priority = 'normal'
resolution = None
stage = 'patch review'
status = 'open'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue17852'
versions = ['Python 3.6', 'Python 3.7']

Linked PRs

GH-62052: add defer_close logic to FileIO #105171

In Python 2, a buffered file opened for writing is flushed by the C library when the process exit. In Python 3, the _pyio and _io modules don't do it reliably. They rely on __del__ being called, which is not neccesarily the case. The attached example ends with the file 'foo' empty (tested with Python 3.3 and the current trunk). The same example in Python 2 using the built-in open() cannot end with 'foo' empty.

There are actually several issues here:

collection of globals at shutdown is wonky: you should add an explicit "del a,f; gc.collect()" at the end of the script
order of tp_clear calls, or another issue with TextIOWrapper: if you open the file in binary mode ("f = open('...', 'wb'); f.write(b'bar')"), the data gets flushed during the GC run

Code relying on garbage collection/shutdown hook to flush files is borked:

it's never been guaranteed by the Python specification (neither does Java/C#...)
even with an implementation based on C stdio streams, streams won't get flushed in case of _exit()/abort()/asynchronous signal

Furthermore, I think it's inherently unsafe: in a multi-threaded program, flushing requires locking, and doing this at shutdown is a recipe for deadlock/arbitrary delay (image you have a daemon thread currently holding the lock, e.g. for stdout).

In short, that's an application bug to me.

It used to be a consistently reliable behavior in Python 2 (and we made it so in PyPy too), provided of course that the process exits normally; but it no longer is in Python 3. Well I can see the reasons for not flushing files, if it's clearly documented somewhere as a change of behavior from Python 2.

However I'm complaining about the current behavior: files are flushed *most of the time*. That's a behavior that is clearly misleading, or so I would think. I'm rather sure that there are many small scripts and large programs out there relying on automatic flushing, and then one day they'll hit a case where the file is not flushed and get the worst kind of error: a file unexpectedly truncated at 99% of its length, in a way that cannot be reproduced by small examples.

Feel free to close anyway as not-a-bug; I won't fight the Python 3 behavior, because Python 2 works as expected.

It used to be a consistently reliable behavior in Python 2 (and we made it so in PyPy too), provided of course that the process exits normally; but it no longer is in Python 3. Well I can see the reasons for not flushing files, if it's clearly documented somewhere as a change of behavior from Python 2.

When you say Python 2, I assume you mean CPython 2, right?
Because - AFAICT - files got flushed only by accident, not by design.
For example, I suspect that Jython doesn't flush files on exit (since
the JVM doesn't), and I guess IronPython neither.

However I'm complaining about the current behavior: files are flushed *most of the time*.

That's the problem with implementation-defined behavior ;-)

"In Python 2, a buffered file opened for writing is flushed by the C library when the process exit."

"When you say Python 2, I assume you mean CPython 2, right?
Because - AFAICT - files got flushed only by accident, not by design."

It looks to be a feature of the standard C library, at least the GNU libc. Its libio library installs an exit handler flushing all open files. You can see it if you set a breaking on write() using gdb:

Breakpoint 5, write () at ../sysdeps/unix/syscall-template.S:82
82 ../sysdeps/unix/syscall-template.S: Aucun fichier ou dossier de ce type.

(gdb) where
#0 write () at ../sysdeps/unix/syscall-template.S:82
#1 0xb7e33ba5 in _IO_new_file_write (f=0x8254ef0, data=0x81f9798, n=3) at fileops.c:1289
#2 0xb7e33a84 in new_do_write (fp=0x8254ef0, data=0x81f9798 "bar\267\300\207\366\267\220\227\037\b\220\227\037\b\004d", to_do=3) at fileops.c:543
#3 0xb7e350fe in _IO_new_do_write (fp=0x8254ef0, data=0x81f9798 "bar\267\300\207\366\267\220\227\037\b\220\227\037\b\004d", to_do=3) at fileops.c:516
#4 0xb7e354b5 in _IO_new_file_overflow (f=0x8254ef0, ch=-1) at fileops.c:894
#5 0xb7e36c40 in _IO_flush_all_lockp (do_lock=0) at genops.c:849
#6 0xb7e36d8e in _IO_cleanup () at genops.c:1010
#7 0xb7df5f92 in __run_exit_handlers (status=0, listp=0xb7f683e4, run_list_atexit=true) at exit.c:91
#8 0xb7df5fdd in __GI_exit (status=0) at exit.c:100
#9 0xb7ddc4db in __libc_start_main (main=0x8058f90 <main>, argc=2, ubp_av=0xbffff1b4, init=0x8156960 <__libc_csu_init>, fini=0x81569d0 <__libc_csu_fini>, rtld_fini=0xb7fed280 <_dl_fini>, stack_end=0xbffff1ac)
at libc-start.c:258
#10 0x08058fd1 in _start ()

Source of _IO_flush_all_lockp() in the GNU libc:

http://sourceware.org/git/?p=glibc.git;a=blob;f=libio/genops.c;h=390d8d24b5fb04f6a35b8fec27e700b9a9d641d4;hb=HEAD#l816

So the glibc maintains a list of open "FILE*" files: _IO_list_all, which is protected by list_all_lock lock.

"When you say Python 2, I assume you mean CPython 2, right?
Because - AFAICT - files got flushed only by accident, not by design."

It looks to be a feature of the standard C library, at least the GNU libc. Its libio library installs an exit handler flushing all open files. You can see it if you set a breaking on write() using gdb:

Yes, it's guaranteed by POSIX/ANSI (see man exit).
I was refering to the fact that the automatic flushing of files upon
exit is a mere side effect of the implementation based atop stdio
stream in cpython 2. It's no guaranteed by any Python spec (and I
can't really think of any platform other than C that makes such
guarantee).

> It looks to be a feature of the standard C library, at least the GNU libc.
Yes, it's guaranteed by POSIX/ANSI (see man exit).

Hum, POSIX (2004) is not so strict:
"Whether open streams are flushed or closed, or temporary files are
removed is implementation-defined."
http://pubs.opengroup.org/onlinepubs/009695399/functions/exit.html

2013/4/29 Charles-François Natali <report@bugs.python.org>:

Charles-François Natali added the comment:

> "When you say Python 2, I assume you mean CPython 2, right?
> Because - AFAICT - files got flushed only by accident, not by design."
>
> It looks to be a feature of the standard C library, at least the GNU libc. Its libio library installs an exit handler flushing all open files. You can see it if you set a breaking on write() using gdb:

Yes, it's guaranteed by POSIX/ANSI (see man exit).
I was refering to the fact that the automatic flushing of files upon
exit is a mere side effect of the implementation based atop stdio
stream in cpython 2. It's no guaranteed by any Python spec (and I
can't really think of any platform other than C that makes such
guarantee).

----------

Python tracker <report@bugs.python.org>
<http://bugs.python.org/issue17852\>

Hum, POSIX (2004) is not so strict:
"Whether open streams are flushed or closed, or temporary files are
removed is implementation-defined."
http://pubs.opengroup.org/onlinepubs/009695399/functions/exit.html

You're looking at the wrong section:

"The exit() function shall then flush all open streams with unwritten
buffered data, close all open streams"

Then a couple lines below:

"The _Exit() [CX] and _exit() functions shall not call functions
registered with atexit() nor any registered signal handlers. Whether
open streams are flushed or closed, or temporary files are removed is
implementation-defined."

It's guaranteed for exit(), not _Exit()/_exit().

It's guaranteed for exit(), not _Exit()/_exit().

Oops ok, thanks.

2013/4/30 Charles-François Natali <report@bugs.python.org>:

Charles-François Natali added the comment:

> Hum, POSIX (2004) is not so strict:
> "Whether open streams are flushed or closed, or temporary files are
> removed is implementation-defined."
> http://pubs.opengroup.org/onlinepubs/009695399/functions/exit.html

You're looking at the wrong section:

"The exit() function shall then flush all open streams with unwritten
buffered data, close all open streams"

Then a couple lines below:

"The _Exit() [CX] and _exit() functions shall not call functions
registered with atexit() nor any registered signal handlers. Whether
open streams are flushed or closed, or temporary files are removed is
implementation-defined."

It's guaranteed for exit(), not _Exit()/_exit().

----------

Python tracker <report@bugs.python.org>
<http://bugs.python.org/issue17852\>

In 3.4+ the example script always writes string "bar" to file "foo". Tested by running it in a loop hundreds times. Cleaning up at shutdown was enhanced in 3.4.

Is anyone interested to work on the "maintain a list of open file objects" idea?

I consider that Python 3 does its best to flush data at exit, but it's not a good practice to rely on the destructors to flush data. I mean, there is no warranty that files will be written in the correct order and that files will stay consistent. It's much safer to explicitly call the clode() method.

By the way, Python 3 now emits ResourceWarning warnings when files are destroyed without being explicitly closed. I enhanced the code handling warnings during Python shutdown to show these warnings in more use cases, but it is still not perfect. I opened the issue bpo-21788 as a reminder that the code can still be enhanced.

For the "maintain a list of objects" idea, I expect race conditions with threads, signals and processes (fork). We already have known issues with locks + fork. I expect even more issues if we maintain a list of open files.

I propose to close the issue as "wontfix". Python 3 does its best, but please call explicitly the close() method!

Maybe the undefined behaviour should be documented (with a warning?) in the buffered writer of the io module, and maybe also in the open() function?

I'm +1 on closing this. Agree with Charles-François that it's never been guaranteed by the Python specification. Pythonic way to work with files is to use the "with" statement, or, if you need long living file stream, careful close files in the "finally" block or in __exit__ method of some class which is used as context manager. Non-observance of this rule can be considered as a bug (see bpo-22831).

Since the issue seems to have been active lately, may I suggest my view on solving it.

One solution that comes to my mind is to keep a weak reference to the file, and to register an atexit function per file (and to unregister it when the file is closed). Example concept-illustrating python code for the _pyio module:

import atexit, weakref

# ...

class TextIOWrapper(TextIOBase):
    def __init__(self):
        # ...
        self._weakclose = operator.methodcaller('close')
        atexit.register(self._weakclose, weakref.proxy(self))

# ...

    def close(self):
        atexit.unregister(self._weakclose)
        # and now actually close the file

There is a possibility of a negative impact arising from the use of operator.methodcaller, because it may return something cached in future versions of python. There is also the issue of unregistering an atexit function during atexit processing. But the general idea seems to be simple and worth considering.

Using atexit is not the solution because the data can be lost even while the program is running, not just at interpreter shutdown. The problem occurs if the buffered file object and the underlying file object are both part of a reference cycle. Then, when the cycle is destroyed by the GC module, if the file __del__ is called before the buffer __del__, data is lost.

So far, the best solution I've come up with is as follows: split the buffered file object into two objects, a wrapper object that will be returned from open() and an underlying object that holds the actual buffer. Make the underlying file object keep references to all the buffers that are using the file. When the file _del__ gets called, first flush all of the buffers and then close the file. Splitting the buffered file object is required so that we don't create reference cycles.

I attempted to implement my weakref idea (i.e. raw file keeps a weakref to the buffered file, calls flush before the raw file gets closed). That doesn't work either because the GC clears the weakref before calling __del__.

This may be a bit of a left-field or too-big-a-hammer suggestion, but as far I can tell from this thread [1] it probably is technically possible to modify the GC to clear weakrefs after calling __del__. Nick wasn't a fan (he likes the invariant that weakrefs can't trigger a resurrection), but if all else fails it might be worth re-raising.

You could add a secondary reference count on FileIO recording how many BufferedIO wrappers there are around it; then it's __del__ would skip calling close() if there are still BufferedIO wrappers, and BufferedIO.__del__ would decrement the reference count and close the underlying file if it hits zero and FileIO.__del__ had already been called.

[1] https://mail.python.org/pipermail/python-dev/2016-October/146747.html

I marked bpo-38548 as duplicate of this issue. Copy of msg355062:

Consider the following program:

f = open("out.txt", "w")
f.write("abc\n")
exit(0)

Please note the absence of f.close().
The documentation
https://docs.python.org/3/tutorial/inputoutput.html#reading-and-writing-files
says that you should use f.close() or with f = open(), but is not clear whether the program above without f.close() is guaranteed to write. The tutorial says:
"If you don’t explicitly close a file, Python’s garbage collector will eventually destroy the object and close the open file for you, but the file may stay open for a while. Another risk is that different Python implementations will do this clean-up at different times."
For me this sounds like even without f.close() the file is guaranteed to be written. If it is not guaranteed to be written, you should fix the documentation, if it is guaranteed to be written, then I will open another issue because the following program does not write into out.txt on my machine:

from sympy.core import AtomicExpr
class MyWeirdClass(AtomicExpr):
	def __init__(self):
		pass
f = open("out.txt", "w")
f.write("abc\n")
exit(0)

Note: sys.version is: "3.7.3 (default, Oct 7 2019, 12:56:13) \n[GCC 8.3.0]"

New changeset c8aaf71 by Volker-Weissmann in branch 'master':
bpo-17852: Doc: Fix the tutorial about closing files (GH-23135)
c8aaf71

New changeset 01fcde8 by Miss Islington (bot) in branch '3.8':
bpo-17852: Doc: Fix the tutorial about closing files (GH-23135)
01fcde8

The PR below fixed it:

#23135

New changeset 4a44f53 by Miss Islington (bot) in branch '3.9':
[3.9] bpo-17852: Doc: Fix the tutorial about closing files (GH-23135) (GH-23527)
4a44f53

is there still an issue to be fixed here? if so re-open and explain or open a new issue. The docmentation was updated to highlight that unclose files can't be guaranteed to flush buffered data.

I think there is still an issue here, even though it is a low priority bug. It's good we document now that unclosed files could lose buffered data. However, it is bad, IMHO, that it works for 99%+ of the time but sometimes doesn't. I think that's worth fixing, if we can do it without too much pain.

I created GH-105171 as a different approach to fix this.

Over last years, I made many enhancements to be able to log ResourceWarning as late as possible. But Python finalization is not deterministic and so some warnings cannot be logged when emitted "too late".

My latest failed attempt to make the finalization more determistic: #86837

The root cause is the GC algorithm for breaking reference cycles. If we have a cycle:

flowchart LR;
    A==>B;
    B==>C;
    C==>A;
    C-->TextIOWrapper;
    TextIOWrapper-->BufferedWriter;
    BufferedWriter-->FileIO;

then it will try to break it by breaking any reference, including references C-->TextIOWrapper, TextIOWrapper-->BufferedWriter and BufferedWriter-->FileIO, even if they do not lead to breaking the cycle. The only references which advance in reference cycle elimination are A-->B, B-->C and C-->A. If the algorithm be smarter, this would solve most of problems, not only with IO.

Navigation Menu

Repository navigation

Built-in module _io can lose data from buffered files in reference cycles #62052

Linked PRs

55 remaining items

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Built-in module _io can lose data from buffered files in reference cycles #62052

Description

Linked PRs

Activity

arigo commented on Apr 27, 2013

pitrou commented on Apr 27, 2013

neologix commented on Apr 27, 2013

arigo commented on Apr 27, 2013

neologix commented on Apr 28, 2013

vstinner commented on Apr 29, 2013

neologix commented on Apr 29, 2013

vstinner commented on Apr 29, 2013

neologix commented on Apr 30, 2013

vstinner commented on Apr 30, 2013

serhiy-storchaka commented on Nov 19, 2014

vstinner commented on Nov 19, 2014

serhiy-storchaka commented on Nov 19, 2014

55 remaining items

Arusekk commented on Jan 24, 2018

nascheme commented on Jan 25, 2018

njsmith commented on Jan 25, 2018

vstinner commented on Oct 21, 2019

methane commented on Nov 27, 2020

miss-islington commented on Nov 27, 2020

Volker-Weissmann commented on Nov 27, 2020

miss-islington commented on Dec 1, 2020

gpshead commented on May 26, 2023

nascheme commented on May 31, 2023

vstinner commented on Jun 1, 2023

serhiy-storchaka commented on Jun 6, 2023

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions