Skip to content

Exchange slowdown #64

Open
Open

Description

@lterrac

We are experiencing slowdowns in the exchange operation used in HiCR. This happens for both MPI and LPF backends.

I have a minimal working example to start troubleshooting the issue.

The part of the example where the slowdown happens is in the createChannels(). In such functions, for N instances we are creating N channels, to organize the instances into a channel ring. To have an idea of the slowdown:

  • 3 instances: 4 seconds
  • 6 instances: 21 seconds
  • 7 instances: 38 seconds

We are currently running in a single node setup with multiple ranks. Timings are similar for lpfThread and mpiThread.

Setup and compile:

git submodule update --init
meson setup build -Dengines=mpi,lpf -DbuildExamples=true -DbuildTests=true
meson compile -C build

Run examples:

mpirun -n 3 --oversubscribe env LPF_ENGINE=zero build/examples/telephoneGame/lpfThread
mpirun -n 3 --oversubscribe build/examples/telephoneGame/mpiThread

Activity

  1. lterrac commented on Oct 7, 2025

    @lterrac
    Author
  2. anyzelman commented on Oct 7, 2025

    @anyzelman
    Member

    I quickly looked over the example and the number of channels created is linear, not quadratic. Assuming (to be checked) that the number of calls to lpf_register_global thus is linear, there should not be a quadratic run-time overhead according to the LPF performance semantics-- yet this is what the timings indicate. TBC and then delved deep on soon.

  3. lterrac commented on Oct 7, 2025

    @lterrac
    Author

    @anyzelman yes, the number of channels is linear. Build instruction are either in the first message of this PR, or alternatively at the root of the channel creator repo link

  4. anyzelman commented on Oct 8, 2025

    @anyzelman
    Member

    Gathering comments so far-- the issue:

    • disappears when the run is over P separate nodes, instead of having P processes on a single node;
    • has been tracked down to a MPI collective operation (I suppose this one: https://github.com/Algebraic-Programming/LPF/blob/master/src/MPI/ibverbs.cpp#L512);
    • occurs with OpenMPI (various versions), has not been tested or observed with other MPIs;
    • is not on a critical path that requires resolution (but we nonetheless retain it in case it becomes more relevant or urgent later).

    Given the information so far, couple of questions @lterrac :

    • it is correct this happens when multiple processes are co-located, correct? (These are not multiple threads, these are multiple proceses?)
    • I notice oversubscribe as an argument to mpirun: this leads me to suspect that perhaps the three processes are assigned to the same core? (If using slurm, are you sure you have multiple cores and that there are P slots that each go to their own core? Because if the answer was yes, then oversubscribe should not be necessary -- which leads me to believe that maybe these processes are indeed going to the same core)
  5. lterrac commented on Oct 8, 2025

    @lterrac
    Author
    1. yes: multiple ranks, same machine
    2. I forgot to take it out of the command. I used it when I was testing the channel creator frontend with huge number of ranks (btw, super slow but works 😄). But neither Kiril nor Sergio were using it with the flag.

    P.s. I do not recall seeing the scenario you are describing for 2) (i.e., N ranks mapped to the same core). When I was running the tests on Sergio's workstation I always saw full utilization of the machine cores first. But maybe I am mistaken, I can do a quick run tomorrow

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions