You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
The part of the example where the slowdown happens is in the createChannels(). In such functions, for N instances we are creating N channels, to organize the instances into a channel ring. To have an idea of the slowdown:
3 instances: 4 seconds
6 instances: 21 seconds
7 instances: 38 seconds
We are currently running in a single node setup with multiple ranks. Timings are similar for lpfThread and mpiThread.
I quickly looked over the example and the number of channels created is linear, not quadratic. Assuming (to be checked) that the number of calls to lpf_register_global thus is linear, there should not be a quadratic run-time overhead according to the LPF performance semantics-- yet this is what the timings indicate. TBC and then delved deep on soon.
@anyzelman yes, the number of channels is linear. Build instruction are either in the first message of this PR, or alternatively at the root of the channel creator repo link
occurs with OpenMPI (various versions), has not been tested or observed with other MPIs;
is not on a critical path that requires resolution (but we nonetheless retain it in case it becomes more relevant or urgent later).
Given the information so far, couple of questions @lterrac :
it is correct this happens when multiple processes are co-located, correct? (These are not multiple threads, these are multiple proceses?)
I notice oversubscribe as an argument to mpirun: this leads me to suspect that perhaps the three processes are assigned to the same core? (If using slurm, are you sure you have multiple cores and that there are P slots that each go to their own core? Because if the answer was yes, then oversubscribe should not be necessary -- which leads me to believe that maybe these processes are indeed going to the same core)
I forgot to take it out of the command. I used it when I was testing the channel creator frontend with huge number of ranks (btw, super slow but works 😄). But neither Kiril nor Sergio were using it with the flag.
P.s. I do not recall seeing the scenario you are describing for 2) (i.e., N ranks mapped to the same core). When I was running the tests on Sergio's workstation I always saw full utilization of the machine cores first. But maybe I am mistaken, I can do a quick run tomorrow
Activity
lterrac commentedon Oct 7, 2025
@anyzelman
anyzelman commentedon Oct 7, 2025
I quickly looked over the example and the number of channels created is linear, not quadratic. Assuming (to be checked) that the number of calls to
lpf_register_globalthus is linear, there should not be a quadratic run-time overhead according to the LPF performance semantics-- yet this is what the timings indicate. TBC and then delved deep on soon.lterrac commentedon Oct 7, 2025
@anyzelman yes, the number of channels is linear. Build instruction are either in the first message of this PR, or alternatively at the root of the channel creator repo link
anyzelman commentedon Oct 8, 2025
Gathering comments so far-- the issue:
Given the information so far, couple of questions @lterrac :
oversubscribeas an argument tompirun: this leads me to suspect that perhaps the three processes are assigned to the same core? (If using slurm, are you sure you have multiple cores and that there arePslots that each go to their own core? Because if the answer was yes, then oversubscribe should not be necessary -- which leads me to believe that maybe these processes are indeed going to the same core)lterrac commentedon Oct 8, 2025
P.s. I do not recall seeing the scenario you are describing for 2) (i.e., N ranks mapped to the same core). When I was running the tests on Sergio's workstation I always saw full utilization of the machine cores first. But maybe I am mistaken, I can do a quick run tomorrow