Repository navigation
Combine dataclass exec calls #109870
Description
Feature or enhancement
Proposal:
As mentioned here, dataclass definition can possibly be sped up by combining the multiple exec calls into one.
I have two versions, and I found that this results in either a 1.1-1.22 1.26, or 1.37 speedup versus main (old time / new time)
version 1 combines the exec calls by passing a list of name/txt/locals from each function definition. (see below for open pr)
version 2 simplifies the final exec call to avoid __create_fn__, but I'm not experienced to know if that is safe.
@ericvsmith Let me know if this is a suitable change, and if so, which version we should open a pr based on.
Thank you!
Has this already been discussed elsewhere?
No response given
Links to previous discussion of this feature:
https://discuss.python.org/t/improving-dataclasses-startup-performance/15442/18
Activity
ssweber commentedon Sep 25, 2023
I measured the difference with this benchmark which includes order=True.
Since we are combining exec calls, the speed up over main will be less if order=False
ssweber commentedon Sep 25, 2023
This could be seen as complementary to Caching dataclass, but this proposal shouldn’t use more memory than the status quo. (we arnt storing a cache in the class)
ssweber commentedon Sep 26, 2023
I updated my numbers. I had missed _hash_add, and my simplified exec without the
__create_fn__call was naive and didn't work 😅The new numbers 1.1-1.18 speedup, depending on how many methods are generated (eg, order=True, frozen=True, etc).
I didn't take the time to refactor _hash_add to be included with the other functions, so its exec is still separate. If it was refactored to be included, we could once again bump those numbers up a bit.
ericvsmith commentedon Sep 30, 2023
@ssweber : As I said elsewhere, I think this is a good idea. Thanks for the work on it! I plan on looking at this during the core dev sprint next week.
ssweber commentedon Oct 1, 2023
Glad I can help in some way! I cleaned up and created a pull request. I tried to not change to much, mainly just to prove the modest speedup.
ericvsmith commentedon Oct 13, 2023
I have a PR that's almost ready for this. I got about a 15% speedup for creating the most basic of dataclasses, which adds 3 methods:
__init__,__repr__, and__eq__. I expect the speedup to be faster if adding comparison functions, etc.ssweber commentedon Oct 15, 2023
It's like 1.22 when you include order/frozen! 😄 love the new
FuncBuilderclass.gh-109870: Dataclasses: batch up exec calls (gh-110851)
Squashed commit of the following:
Squashed commit of the following:
pythongh-109870: Dataclasses: batch up exec calls (pythongh-110851)