Skip to content

shutil.copy() inefficient implementation in Windows #88745

Closed
Closed
shutil.copy() inefficient implementation in Windows#88745
Assignees
zooba
Labels
3.12only security fixesOS-windowsperformancePerformance or resource usage

Description

@sfmc
sfmc
mannequin
opened on Jul 7, 2021
BPO 44579
Nosy @pfmoore, @tjguk, @zware, @eryksun, @zooba

Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2021-07-07.12:11:29.301>
labels = ['3.8', 'OS-windows', 'performance']
title = 'shutil.copy() inefficient implementation in Windows'
updated_at = <Date 2021-07-07.14:39:41.523>
user = 'https://bugs.python.org/sfmc'

bugs.python.org fields:

activity = <Date 2021-07-07.14:39:41.523>
actor = 'eryksun'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Windows']
creation = <Date 2021-07-07.12:11:29.301>
creator = 'sfmc'
dependencies = []
files = []
hgrepos = []
issue_num = 44579
keywords = []
message_count = 3.0
messages = ['397076', '397077', '397091']
nosy_count = 6.0
nosy_names = ['paul.moore', 'tim.golden', 'zach.ware', 'eryksun', 'steve.dower', 'sfmc']
pr_nums = []
priority = 'normal'
resolution = None
stage = None
status = 'open'
superseder = None
type = 'performance'
url = 'https://bugs.python.org/issue44579'
versions = ['Python 3.8']

Linked PRs

Activity

  1. sfmc commented on Jul 7, 2021

    @sfmc
    sfmcmannequin
    on Jul 7, 2021
    MannequinAuthor

    In Windows shutil.copy() uses _copyfileobj_readinto which copies file in user mode.
    In Windows there is an fast API to copy file in kernel mode: CopyFile (see https://docs.microsoft.com/en-us/windows/win32/api/winbase/nf-winbase-copyfile).

  2. sfmc
    added
    stdlibStandard Library Python modules in the Lib/ directory
    3.8 (EOL)end of life
    performancePerformance or resource usage
    on Jul 7, 2021
  3. tirkarthi
    added
    OS-windows
    on Jul 7, 2021

  4. eryksun commented on Jul 7, 2021

    @eryksun
    eryksun
    on Jul 7, 2021
    Contributor

    In Windows there is an fast API to copy file in kernel mode: CopyFile

    The possibility of calling CopyFileEx() for shutil.copy2() is discussed in bpo-30044. Note that CopyFileEx() is a high-level Windows API function, not a "kernel mode" copy. It opens the source and destination files and makes multiple system calls in order to copy file data and metadata (e.g. system calls such as NtOpenFile, NtCreateFile, NtReadFile, NtWriteFile, NtQueryInformationFile, NtSetInformationFile, NtQueryEaFile, NtSetEaFile, NtQuerySecurityObject, etc). This includes copying the primary data stream, alternate data streams, file attributes, extended file attributes, and security resource attributes.

  5. eryksun
    removed
    OS-windows
    on Jul 7, 2021

  6. zooba commented on Jul 7, 2021

    @zooba
    zooba
    on Jul 7, 2021
    Member

    Note that CopyFileEx() is a high-level Windows API function, not a "kernel mode" copy.

    This is true today, but could change whenever Windows feels like changing it. If we switch to the native API then we'll get any advantage there automatically.

    The only challenge is in managing changes to semantics (that is, anything extra we do to "match" Unix that isn't normally how copies work on Windows - personally, I'd rather be more native on Windows anyway).

  7. eryksun
    added
    OS-windows
    and removed
    stdlibStandard Library Python modules in the Lib/ directory
    on Jul 7, 2021
  8. ezio-melotti
    transferred this issue fromon Apr 10, 2022
  9. iritkatriel
    added
    3.12only security fixes
    and removed
    3.8 (EOL)end of life
    on Sep 7, 2022
  10. zooba
    self-assigned this
    on May 27, 2023

  11. zooba commented on May 27, 2023

    @zooba
    zooba
    on May 27, 2023
    Member

    I think it's time to do this one. I've assigned it to myself, but if someone else gets an implementation together first (it could use a CopyFileEx implementation in _winapi) I'm happy to review and merge.

    Performance enhancement can be backported to 3.12. The behaviour of copy2 should be totally unchanged by this, except in obscure edge cases (where they'll likely be more consistent with every other Windows app as a result of the change).

    We should also clearly document that copy2 may be significantly faster on Windows than the other copy* functions. Probably also worth looking for optimisations on other platforms as well, particularly if they preserve stat, so that we can say that copy2 is fastest on most platforms.

  12. zooba
    added 2 commits that reference this issue on May 29, 2023

    pythongh-88745: Add _winapi.CopyFile2 and update shutil.copy2 to use it

    dffb615

    gh-88745: Add _winapi.CopyFile2 and update shutil.copy2 to use it (GH…

    cda1bd3
  13. miss-islington
    added a commit that references this issue on May 30, 2023

    pythongh-88745: Add _winapi.CopyFile2 and update shutil.copy2 to use …

    6edc59a
  14. zooba
    added a commit that references this issue on May 30, 2023

    gh-88745: Add _winapi.CopyFile2 and update shutil.copy2 to use it (GH…

    9ae49e3
  15. zooba
    closed this as completedon May 30, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

3.12only security fixesOS-windowsperformancePerformance or resource usage

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions