Activity

1. eMeena

   added

   type-bugAn unexpected behavior, bug, or errorAn unexpected behavior, bug, or error

   on Oct 19, 2022
2. iritkatriel commented on Oct 20, 2022

   

   iritkatriel

   on Oct 20, 2022

   Member

   More actions

   The process_user function returns the allocated memory. Don’t you want the return value to be available in the main thread?

   👍React with 👍1Reacted by Sohang Chopra
3. eMeena commented on Oct 20, 2022

   

   eMeena

   on Oct 20, 2022

   Author

   More actions

   Correct, it should be available in the main thread. The primary issue is that allocated memory is never freed though. It should be freed once futures goes out of scope or when the executor is shutdown which should occur when exciting the with block but that doesn't happen.
4. thomasyang18 commented on Oct 31, 2022

   

   thomasyang18

   on Oct 31, 2022

   Last edited by thomasyang18

   More actions

   This isn't even because of threads, this is normal python behavior:

   import resource
   
   def leak_mem():
       [bytearray(10000000) for i in range(0, 100)]
       return None
   
   print('Before ', resource.getrusage(resource.RUSAGE_SELF).ru_maxrss/1024, 'MB')
   
   leak_mem()
   
   print('After', resource.getrusage(resource.RUSAGE_SELF).ru_maxrss/1024, 'MB')
   

   For example, this fib program "leaks"

   import resource
   
   def get_fib():
       fib = [1, 1]
       for i in range(0, 10**7):
           fib.append((fib[-1] + fib[-2])%100000000)
       return None # never using fib
   
   print('Before ', resource.getrusage(resource.RUSAGE_SELF).ru_maxrss/1024, 'MB')
   
   x = get_fib()
   
   print('After', resource.getrusage(resource.RUSAGE_SELF).ru_maxrss/1024, 'MB')
   

   Editing because I thought that python doesn't destroy on destructor scope. It does (both the threaded example and my example) if you write:

   def process(i):
       bytearray(10000000)
       return None
   

   👍React with 👍1Reacted by Sohang Chopra
5. JelleZijlstra commented on Oct 31, 2022

   

   JelleZijlstra

   on Oct 31, 2022

   Member

   More actions

   I believe the objects are garbage collected, but the memory isn't necessarily returned to the OS because Python uses a layer of abstraction around the system allocator. So you won't see process-level memory usage going back to the previous level.

   However, if you keep doing the same thing, memory usage should stabilize and not keep growing.

   ❤️React with ❤️2Reacted by Yi Xing and Vinod Ankith Devanaboina
6. iritkatriel

   added

   stdlibStandard Library Python modules in the Lib/ directoryStandard Library Python modules in the Lib/ directory

   on Nov 26, 2023
7. nh2 commented on Mar 26, 2025

   

   nh2

   on Mar 26, 2025

   More actions

   @eMeena Your benchmark doesn't make sense: You're using ru_maxrss, which is the maximum memory usage (maximum over the lifetime of the program). It thus can never go down, as it does not measure the current memory usage.

   Thus the print

   print('After', resource.getrusage(resource.RUSAGE_SELF).ru_maxrss/1024, 'MB')

   makes no sense -- that's expected to be, and stay, high.

   To get the current memory usage, you need to measure rss, not maxrss. For example this way with psutil.Process().memory_info().rss.

   (Note I'm also coming here because my ThreadPoolExecutor.map() seems to somehow memory leak, but the method from the issue description surely isn't the way to measure that.)
8. d33bs commented on May 20, 2025

   

   d33bs

   on May 20, 2025

   More actions

   I got curious about this because of my work through Parsl's ThreadPoolExecutor, which from my understanding takes advantage of CPython's ThreadPoolExecutor (Parsl/parsl#2876, cytomining/CytoTable#75). Below, I borrowed @eMeena's example and implemented @nh2's suggestion with the following to try and demonstrate what @JelleZijlstra mentioned.

   """
   Show how ThreadPoolExecutor might cause memory challenges
   """
   
   import os
   import psutil
   import gc
   import random
   
   from concurrent.futures import ThreadPoolExecutor, as_completed
   
   if __name__ == "__main__":
       # one-time process handle
       _proc = psutil.Process(os.getpid())
   
       def current_rss_mb() -> float:
           # show resident memory in megabytes
           return _proc.memory_info().rss / (1024**2)
   
       def do_something(x):
           # generate random data
           return [random.randint(0, 2**31 - 1) for _ in range(5_000)]
   
       def pool_stuff():
           with ThreadPoolExecutor(max_workers=20) as executor:
               futures = [executor.submit(do_something, i) for i in range(100)]
               for i, future in enumerate(as_completed(futures), 1):
                   # force retrieval to ensure result doesn’t stick around
                   _ = future.result()
   
       print("Before:", current_rss_mb(), "MB")
   
       for i in range(10):
           # run the exectutor tasks
           pool_stuff()
   
           # show memory usage
           print(f"result {i}: ", current_rss_mb(), "MB")
   
       # collect garbage
       gc.collect()
   
       print("Finally: ", current_rss_mb(), "MB")

   When I run this on MacOS with Python 3.12 I see something like the following (varies each time):

   Before: 15.75 MB
   result 0:  30.75 MB
   result 1:  31.734375 MB
   result 2:  32.71875 MB
   result 3:  33.015625 MB
   result 4:  33.765625 MB
   result 5:  33.953125 MB
   result 6:  34.421875 MB
   result 7:  34.421875 MB
   result 8:  34.4375 MB
   result 9:  34.484375 MB
   Finally:  26.484375 MB
   

   It still seems like there's a gradual increase each time we run ThreadPoolExecutor, most of the time. Afterwards, we seem to regain about a quarter of our memory used during processing if we suggest garbage collection. Tinkering around with this we seem to regain roughly the same amount of memory each time from garbage collection. It still has me wondering about how to get back to the "before" state.