Skip to content

Flowable.groupBy hangs with group consumed via observeOn and take(timed) #6982

Closed

Description

@akarnokd

Continuing evaluating the problems from #6974

The following test hangs after a while:

    @Test
    public void issue6974Part2Case1ObserveOnNoCap() {
        final int groups = 20;
        int flatMapMaxConcurrency = 1_000_000;

        Flowable
        .range(1, 500_000)
        .map(i -> i % groups)
        .doOnRequest(v -> {
            System.out.println("Source: " + v);
        })
        .groupBy(i -> i)
        .flatMap(gf -> gf
                     .observeOn(Schedulers.computation())
                     .take(10, TimeUnit.MILLISECONDS)
            , flatMapMaxConcurrency)
        .test()
        .awaitDone(5, TimeUnit.SECONDS)
        .assertNoErrors()
        .assertComplete();
    }

Adding hide makes it pass which indicates the fusion between the group and observeOn seems to stop requesting more items.

Activity

  1. added this to the 3.1 milestone on May 13, 2020
  2. akarnokd commented on May 13, 2020

    @akarnokd
    MemberAuthor

    Thoughs on the cause so far:

    Unconditional group removal

    The unconditional group removal in cancel(K key) could be a contributing factor; debug indicated 0 groups during the hang.

    Rejected execution

    take makes observeOn dispose its worker so items are stuck in the (fused) queue not being cleaned.

    Currently, there is no good way to detect such rejection through the Scheduler.Worker API as rejected schedules return EmptyDisposable.INSTANCE and TrampolineScheduler also returns it as a default indicator. This would require API changes and the introduction of a dedicated and public rejected-disposable.

    Second, observeOn has to be updated to detect rejection and perform the cleanup on the fused queue.

    Third, the cancellation can come after the schedule but before the drain execution, thus the drain is never executed and the operator is stuck in wip > 0 state.

    Difference with hide()

    With hide(), the group manages the items and can request more upon cancellation, hence there should be no hanging.

  3. sobersanta commented on May 13, 2020

    @sobersanta
    Contributor

    My observations with the latest version:

    1. when evicting groups from the map externally (ttl, size, etc) without async boundary downstream, hangs sometimes on the very last item, not every time.

    Test case:

    Flowable
      .range(1, 500_000)
      .map(i -> i % 20)
      .groupBy(i -> i, i -> i, false, groupByBufferSize,
                             sizeAndTtlCapGuava(1000, Duration.ofMillis(10), false))
      .flatMap(gf -> gf, groups)
      .test()
      .awaitDone(5, TimeUnit.SECONDS)
      .assertComplete();
    1. when cancelling groups e.g. by take() operator.

    Here we have both issue with hangs and abandonment.

    Please try

    Flowable
      .range(1, 500_000)
      .map(i -> i % 20)
      .groupBy(i->i) // no any group eviction
      .flatMap(gf -> gf.take(10)
                                     .observeOn(Schedulers.computation())
                        ,groups)
      .test()
      .awaitDone(5, TimeUnit.SECONDS)
      .assertComplete();
  4. sobersanta commented on May 13, 2020

    @sobersanta
    Contributor
    1. same as 1) but with delayed oncomplete : in this case delayed oncomplete->request leads to MBE
    Flowable
                    .range(1, 500_000)
                    .map(i -> i % groups)
                    .groupBy(i -> i, i -> i, false, groupByBufferSize,
                             sizeAndTtlCapGuava(1000, Duration.ofMillis(10), false))
                    .flatMap(gf -> gf
                                 .observeOn(Schedulers.computation())
                        , groups)
                    .test()
                    .awaitDone(5, TimeUnit.SECONDS)
                    .assertComplete();
  5. akarnokd commented on May 13, 2020

    @akarnokd
    MemberAuthor

    We can't do anything about the MBEs, it is inherent when running with limited concurrency because a request from one group can lead to a new group creation for which if there is no request on the main output, there is no way to store it in a bounded manner.

    The objective here is to not let the chain hang but have it complete or fail with MBE depending on the situation.

  6. sobersanta commented on May 13, 2020

    @sobersanta
    Contributor

    This lack of coordination between group eviction/consequent flatMap's request greatly limits use cases of the operator :(
    Maybe there could be considered some change in it's design so it can become possible?
    E.g. queue up emissions of new groups, caused specifically by eviction (it's relatively easy to detect) until there is a request from downstream?
    Or do emissions on the request, not during onNext?

  7. sobersanta commented on May 13, 2020

    @sobersanta
    Contributor

    The question is that there is a time gap between completion of a group and consequent request, while if it was synchronously, there wouldn't be an issue with MBE.
    Issue looks exactly the same as documented requirement of a mandatory synchronous subscription to a group right after emitted.

  8. sobersanta commented on May 13, 2020

    @sobersanta
    Contributor

    OK Here is another thought .... Which will demonstrate that Hang issues are related to MBEs.

    In the request method there is only an increment of the atomic counter.
    I can't find any signs of informing upstream about the request.
    The only place where request upstream happens, is during synchronous subscription - flatMap immediately prefetches upstream.
    That request is a basis for all upcoming groupBy operator work.
    And if there is a delay between an emission of a group and replenishing request from the downstream, there is a high possibility of a hang / MBE because noone is coordinating emissions with requests, and also noone is informing upstream about a request from downstream - if emission downstream requests do not catch up with the emissions.

    Makes sense?

  9. sobersanta commented on May 13, 2020

    @sobersanta
    Contributor

    Please note I'm talking about requests from the flatMap, not from the consumers of the groups!

  10. sobersanta commented on May 18, 2020

    @sobersanta
    Contributor

    Hi. Really don't want to bother you, but could you please let us know, what are the plans regarding groupBy operator?

  11. sobersanta commented on May 18, 2020

    @sobersanta
    Contributor

    Latest snapshot still hangs for me with these tests:
    First one hangs in around 50% of runs always exactly at the last item, second hangs always.

    import com.google.common.cache.CacheBuilder;
    import io.reactivex.rxjava3.core.Flowable;
    import io.reactivex.rxjava3.functions.Consumer;
    import io.reactivex.rxjava3.functions.Function;
    import io.reactivex.rxjava3.schedulers.Schedulers;
    
    import java.time.Duration;
    import java.util.concurrent.ConcurrentMap;
    import java.util.concurrent.TimeUnit;
    
    public class Test {
        public static <T> Function<Consumer<Object>, ConcurrentMap<T, Object>> ttlCapGuava(Duration ttl) {
            return itemEvictConsumer -> CacheBuilder
                                            .newBuilder()
                                            .expireAfterWrite(ttl)
                                            .removalListener(n -> {
                                                if (n.getCause() != com.google.common.cache.RemovalCause.EXPLICIT) {
                                                    try {
                                                        itemEvictConsumer.accept(n.getValue());
                                                    } catch (Throwable throwable) {
                                                        throw new RuntimeException(throwable);
                                                    }
                                                }
                                            }).<T, Object>build().asMap();
        }
    
        public static void main(String[] args) {
            final int groups = 20;
    
            int groupByBufferSize = 2;
            int flatMapMaxConcurrency = 200 * groups;
            // ~50% of executions - Not completed (latch = 1, values = 500000, errors = 0, completions = 0, timeout!,
            // disposed!)
            test(groupByBufferSize, flatMapMaxConcurrency, groups);
    
            groupByBufferSize = groups * 30;
            flatMapMaxConcurrency = groups * 500;
            // Always : Not completed (latch = 1, values = 14100, errors = 0, completions = 0, timeout!, disposed!)
            test(groupByBufferSize, flatMapMaxConcurrency, groups);
        }
    
        public static void test(int groupByBufferSize, int flatMapMaxConcurrency, int groups) {
            try {
                Flowable
                    .range(1, 500_000)
                    .map(i -> i % groups)
                    .groupBy(i -> i, i -> i, false, groupByBufferSize, ttlCapGuava(Duration.ofMillis(10)))
                    .flatMap(gf -> gf.observeOn(Schedulers.computation()), flatMapMaxConcurrency)
                    .test()
                    .awaitDone(5, TimeUnit.SECONDS)
                    .assertComplete();
                System.out.println("Success");
            } catch (Throwable e) {
                e.printStackTrace();
            }
        }
    }
  12. akarnokd commented on May 18, 2020

    @akarnokd
    MemberAuthor

    I'll be looking into them (and a few additional cases I was thinking about the past days) tomorrow.

  13. sobersanta commented on May 18, 2020

    @sobersanta
    Contributor

    Thanks!!!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

      Milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions