Stop manually interning strings in pathlib

When parsing and normalizing a path, pathlib calls sys.intern() on the string parts:

Line 273 in 96b392d

parsed = [sys.intern(str(x)) for x in rel.split(sep) if x and x != '.']

I've never been able to establish that this is a worthwhile thing to do. The implementation seems incomplete, because the path normalization only occurs when a user manually initialises a path object, and not in paths generated from path.iterdir(), path.walk(), etc. Drives/roots/anchors aren't interned despite most likely to be shared.

Previous discussion: #112856 (comment)

Linked PRs

GH-119518: Stop interning strings in pathlib #123356

@pitrou

CC @pitrou

I've never been able to establish that this is a worthwhile thing to do.

That's a good question. The contention is that, if you keep a lot of related paths in memory, interning the path components would yield significant memory savings. But how useful it is would depend on the use case; and it's probably possible to construct use cases where it would be detrimental.

Thank you :)

it's probably possible to construct use cases where it would be detrimental.

Surprisingly, sys.intern(str(x)) accounts for ~10% of the cost of str(Path('foo', 'bar')) on my machine. i.e. this patch makes it ~10% faster:

diff --git a/Lib/pathlib/_local.py b/Lib/pathlib/_local.py
index 49d9f813c5..d20512bd9b 100644
--- a/Lib/pathlib/_local.py
+++ b/Lib/pathlib/_local.py
@@ -270,7 +270,7 @@ def _parse_path(cls, path):
             elif len(drv_parts) == 6:
                 # e.g. //?/unc/server/share
                 root = sep
-        parsed = [sys.intern(str(x)) for x in rel.split(sep) if x and x != '.']
+        parsed = [x for x in rel.split(sep) if x and x != '.']
         return drv, root, parsed
 
     @property

It can probably be micro-optimized if you really care. For example I'm not sure the str call is still useful.

>>> x = 'foo'
>>> %timeit str(x)
40.3 ns ± 0.169 ns per loop (mean ± std. dev. of 7 runs, 10,000,000 loops each)
>>> %timeit sys.intern(x)
46.9 ns ± 0.402 ns per loop (mean ± std. dev. of 7 runs, 10,000,000 loops each)
>>> %timeit sys.intern(str(x))
79.7 ns ± 0.635 ns per loop (mean ± std. dev. of 7 runs, 10,000,000 loops each)
>>> _intern = sys.intern
>>> %timeit _intern(x)
28.3 ns ± 0.11 ns per loop (mean ± std. dev. of 7 runs, 10,000,000 loops each)

Hum! That change used to trip up the test_str_subclass* tests, but it doesn't any more. I wonder if there's been a change in how string subclasses work elsewhere in the language.

The contention is that, if you keep a lot of related paths in memory, interning the path components would yield significant memory savings.

Can pathlib have its own "interned strings" cache with a limit on the cache size? Well, I don't know if it's worth it :-)

Pseudo-code with a limit on 3 entries:

cache = {}
def get(key):
    try:
        return cache[key]
    except KeyError:
        pass

    if len(cache) >= 3:
        return key

    cache[key] = key
    return key

abc = get(b'abc'.decode())
abc2 = get(b'abc'.decode())
assert abc2 is abc

d = get(b'd'.decode())
e = get(b'e'.decode())
e2 = get(b'e'.decode())
assert e2 is e

# cache no longer used
f = get(b'f'.decode())
print("cache size", len(cache))

Can pathlib remove entries from such cache?

pythonGH-119518: Stop interning strings in pathlib

GH-119518: Stop interning strings in pathlib GH-123356)

Navigation Menu

Repository navigation

Stop manually interning strings in pathlib #119518

Linked PRs

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Stop manually interning strings in pathlib #119518

Description

Linked PRs

Activity

barneygale commented on May 24, 2024

pitrou commented on May 28, 2024

barneygale commented on May 28, 2024

pitrou commented on May 28, 2024

barneygale commented on May 28, 2024

vstinner commented on Jun 3, 2024

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions