Skip to content

3.0.3 - Hangs when using combination of flatMap and publish #6974

Description

@sobersanta

Hi.

After few days of desperate attempts to find the cause of my servers' spontaneous hangs, I managed to reproduce the issue, but still don't have a clear idea about the reason, but I think I have identified a possible direction.

All problems has started when I started using very promising operator groupBy with evicting map provider possibility:

<K, V> Flowable<GroupedFlowable<K, V>> groupBy(@NonNull Function<? super T, ? extends K> keySelector,
            @NonNull Function<? super T, ? extends V> valueSelector,
            boolean delayError, int bufferSize,
            @NonNull Function<? super Consumer<Object>, ? extends Map<K, Object>> evictingMapFactory)

First I realised that it doesn't work well with Caffeine cache, as it performs evictions on a separate thread pool, and with async eviction notification groupBy goes crazy.

After moving to Guava things got better but only to some extent.

The tests I want to demonstrate I believe reproduce the issue I have. Definitely chain of operators is way more complicated in the production code, but here with certain variant of .publish() operator it's reproducible as well.

Basically, I'm not sure if there is issue in the groupBy(...evictingMapFactory) or in the publish(Function...), or in both.

Also tests are demonstrating that eviction policy doesn't work very well when crossing async boundaries downstream.

Please take a look at the attached file, it contains few simple tests, I kindly ask to check carefully all parameters, as it's important.

FlowableOperators.java.zip

Activity

  1. akarnokd commented on May 9, 2020

    @akarnokd
    Member

    Hi and thanks for reporting this bug.

    The problem is caused by the eviction double decrementing the group counter, once when the eviction completes the shared chain that cancels the group and once after the eviction.

    I'll post a fix shortly.

  2. sobersanta commented on May 9, 2020

    @sobersanta
    ContributorAuthor

    Cool, looking forward for the fix!

  3. akarnokd commented on May 10, 2020

    @akarnokd
    Member

    /cc @davidmoten regarding the async eviction concerns.

  4. sobersanta commented on May 10, 2020

    @sobersanta
    ContributorAuthor

    Thanks for the prompt reply and the fix!
    May I ask, when we can expect to have it to be released?
    Maybe 3.0.4 soon? ;)

  5. akarnokd commented on May 10, 2020

    @akarnokd
    Member

    I suggest you verify it by testing with the current RxJava snapshot, just in case: https://github.com/ReactiveX/RxJava#snapshots

  6. sobersanta commented on May 10, 2020

    @sobersanta
    ContributorAuthor

    Sure, I will let you know about results.

  7. sobersanta commented on May 10, 2020

    @sobersanta
    ContributorAuthor

    Could you please comment on this?

        public static void main(String[] args) {
    
            final int groups = 20;
    
            // Not completed (Timed out), buffer is too small
            int groupByBufferSize = groups * 2;
            int flatMapMaxConcurrency = 2 * groups;
            boolean notifyOnExplicitEviction = false;
            test2(groupByBufferSize, flatMapMaxConcurrency, groups, notifyOnExplicitEviction);
    
            // Success, buffer is large enough
            groupByBufferSize = groups * 30;
            flatMapMaxConcurrency = 2 * groups;
            test2(groupByBufferSize, flatMapMaxConcurrency, groups, notifyOnExplicitEviction);
    
            // Timeout... explicit eviction notification makes difference
            groupByBufferSize = groups * 30;
            flatMapMaxConcurrency = 2 * groups;
            notifyOnExplicitEviction = true;
            test2(groupByBufferSize, flatMapMaxConcurrency, groups, notifyOnExplicitEviction);
    
            // MissingBackpressureException, flatMap concurrency == groups count is not enough, must be at least 2x
            groupByBufferSize = groups * 20;
            flatMapMaxConcurrency = groups;
            notifyOnExplicitEviction = false;
            test2(groupByBufferSize, flatMapMaxConcurrency, groups, notifyOnExplicitEviction);
        }
    
        public static void test2(int groupByBufferSize, int flatMapMaxConcurrency, int groups,
                                 boolean notifyOnExplicitEviction) {
            try {
                TestSubscriber<Integer> test = Flowable
                    .range(1, 500_000)
                    .map(i -> i % groups)
                    .groupBy(i -> i, i -> i, false, groupByBufferSize,
                             // set cap too high
                             sizeCap(groups * 100, notifyOnExplicitEviction))
                    .flatMap(gf -> gf
                                 .take(10, TimeUnit.MILLISECONDS)
                        , flatMapMaxConcurrency)
                    .test()
                    .awaitDone(5, TimeUnit.SECONDS)
                    .assertComplete();
                System.out.println("Success");
            } catch (Throwable e) {
                e.printStackTrace();
            }
        }
    
        public static <T> Function<Consumer<Object>, Map<T, Object>> sizeCap(int maxCapacity, boolean notifyOnExplicit) {
            return itemEvictConsumer ->
                CacheBuilder
                    .newBuilder()
                    .maximumSize(maxCapacity)
                    .removalListener(n -> {
                        if (n.getCause() != RemovalCause.EXPLICIT || notifyOnExplicit) {
                            try {
                                itemEvictConsumer.accept(n.getValue());
                            } catch (Throwable throwable) {
                                throw new RuntimeException(throwable);
                            }
                        }
                    })
                    .<T, Object>build().asMap();
        }
  8. sobersanta commented on May 10, 2020

    @sobersanta
    ContributorAuthor

    changing .take(10, MILLISECONDS) to .take(10) makes only 3d case failing with timeout

  9. akarnokd commented on May 10, 2020

    @akarnokd
    Member

    Is this with the snapshot?

  10. sobersanta commented on May 10, 2020

    @sobersanta
    ContributorAuthor

    yes, snapshot

  11. 12 remaining items

  12. sobersanta commented on May 11, 2020

    @sobersanta
    ContributorAuthor

    More than clear, thanks for the answer!

  13. sobersanta commented on May 11, 2020

    @sobersanta
    ContributorAuthor

    As for the Map's remove(Object,Object).
    WDYT about checking if the map is instance of ConcurrentMap (I believe it will be like that in most of the cases, I can hardly imagine anyone would uses anything else), and if yes then using ConcurrentMap.remove(key, value), otherwise keep existing logic?
    That check can be done during the construction, and for example keep 2 nullable references to maps, or wrap a non-ConcurrentMap's implementation with a proper delegating wrapper.

  14. sobersanta commented on May 11, 2020

    @sobersanta
    ContributorAuthor

    I have tried the code from your latest PR. And my pipeline still hangs on the graceful completion :(
    So far don't have a simple test which is reproducing the problem, will let you know when I get something. Could be a problem is in another operator. (Though it works fine if I serialize calls to State.cancel and upstream onNext in groupby)

  15. sobersanta commented on May 11, 2020

    @sobersanta
    ContributorAuthor

    OK here is the case (just added .observeOn to grouped flowable):

    public static void test2(int groupByBufferSize, int flatMapMaxConcurrency, int groups,
                                 boolean notifyOnExplicitEviction) {
            try {
                Flowable
                    .range(1, 500_000)
                    .map(i -> i % groups)
                    .doOnCancel(() -> System.out.println("Cancelling upstream"))
                    .compose(s -> groupBy(s, i -> i, i -> i, groupByBufferSize, false,
                                          sizeCap(groups *2, notifyOnExplicitEviction)))
                    .flatMap(gf -> gf
                                 .observeOn(Schedulers.computation())
                                 // .take(10)
                                 .take(10, TimeUnit.MILLISECONDS)
                        , flatMapMaxConcurrency)
                    .test()
                    .awaitDone(5, TimeUnit.SECONDS)
                    .assertComplete();
                System.out.println("Success");
            } catch (Throwable e) {
                e.printStackTrace();
            }
        }
  16. akarnokd commented on May 11, 2020

    @akarnokd
    Member

    If you put hide() before observeOn, does it work then?

  17. sobersanta commented on May 11, 2020

    @sobersanta
    ContributorAuthor

    If you put hide() before observeOn, does it work then?

    Apparently yes

  18. sobersanta commented on May 11, 2020

    @sobersanta
    ContributorAuthor

    I can see that partially my problems are caused by the .publish() operator (it uses FlowablePublishMulticast) which on complete calls .dispose() which in turn cancels upstream. I'm not sure I understand the reason of that canceling while this is a normal graceful shutdown. Could you please elaborate a little for a dummy like me why is it cancelling upstream on complete event? It messes with my resource cleanup logic (onTerminate is never called after that), and I'm not sure how to overrun it.

    BTW here's the case where the new code fails even with .hide() in place:

        public static void test2(int groupByBufferSize, int flatMapMaxConcurrency, int groups,
                                 boolean notifyOnExplicitEviction) {
            try {
                Flowable
                    .range(1, 500_000)
                    .map(i -> i % groups)
                    .doOnCancel(() -> System.out.println("Cancelling upstream"))
                    .compose(s -> groupBy(s, i -> i, i -> i, groupByBufferSize, false,
                                          sizeAndTtlCapGuava(groups * 2, Duration.ofMillis(10), notifyOnExplicitEviction)))
                    .flatMap(gf -> gf
                                 .hide()
                                 .publish(shared -> shared)
                        , flatMapMaxConcurrency)
                    .test()
                    .awaitDone(5, TimeUnit.SECONDS)
                    .assertComplete();
                System.out.println("Success");
            } catch (Throwable e) {
                e.printStackTrace();
            }
        }
    
        public static <T> Function<Consumer<Object>, ConcurrentMap<T, Object>> sizeAndTtlCapGuava(int maxCapacity,
                                                                                                  Duration ttl,
                                                                                                  boolean notifyOnExplicit) {
            return itemEvictConsumer ->
                CacheBuilder
                    .newBuilder()
                    .maximumSize(maxCapacity)
                    .expireAfterWrite(ttl)
                    .removalListener(n -> {
                        if (n.getCause() != com.google.common.cache.RemovalCause.EXPLICIT || notifyOnExplicit) {
                            try {
                                itemEvictConsumer.accept(n.getValue());
                            } catch (Throwable throwable) {
                                throw new RuntimeException(throwable);
                            }
                        }
                    })
                    .<T, Object>build().asMap();
        }
  19. akarnokd commented on May 11, 2020

    @akarnokd
    Member

    why is it cancelling upstream on complete event

    With publish(Function), the inner flow may be independent of the upstream so when the inner terminates, the upstream should be stopped. I suppose when there is a causal chain from an upstream completion to a downstream completion, this cancellation could be suppressed.

    I'll look into the new cases tomorrow.

  20. sobersanta commented on May 12, 2020

    @sobersanta
    ContributorAuthor

    Adding .observeOn leads to MissingbackpressureException:

        public static void main(String[] args) {
            final int groups = 20;
    
            // MissingBackpressureException
            int groupByBufferSize = 2;
            int flatMapMaxConcurrency = 2 * groups;
            boolean notifyOnExplicitEviction = false;
            test2(groupByBufferSize, flatMapMaxConcurrency, groups, notifyOnExplicitEviction);
    
            groupByBufferSize = groups * 30;
            flatMapMaxConcurrency = 2 * groups;
            test2(groupByBufferSize, flatMapMaxConcurrency, groups, notifyOnExplicitEviction);
        }
    
        public static void test2(int groupByBufferSize, int flatMapMaxConcurrency, int groups,
                                 boolean notifyOnExplicitEviction) {
            try {
                Flowable
                    .range(1, 500_000)
                    .map(i -> i % groups)
                    .compose(s -> groupBy(s, i -> i, i -> i, groupByBufferSize, false,
                                          sizeAndTtlCapGuava(groups, Duration.ofMillis(10), notifyOnExplicitEviction)))
                    .flatMap(gf -> FlowableOperators.publish(gf.hide()
                                                               .observeOn(Schedulers.computation())
                        , shared -> shared)
                        , flatMapMaxConcurrency)
                    .test()
                    .awaitDone(5, TimeUnit.SECONDS)
                    .assertComplete();
                System.out.println("Success");
            } catch (Throwable e) {
                e.printStackTrace();
            }
        }
  21. sobersanta commented on May 12, 2020

    @sobersanta
    ContributorAuthor

    Continuing about problem with the latest test above.
    What I realised today is that when a group completes (e.g. because of external eviction from the Map), it pushes down onComplete event, and in case there is observeOn operator downstream, or any other reason which causes onComplete to be scheduled on another thread, the consequence is that expected group request from flatMap (or another operator consuming groups) happens with a delay, which leads to "group abandonment". It's not really about subscribing to a new group only, as stated in the documentation, but requesting more when another one completes.

    Simple reproduction:

    Flowable
      .range(1, 500_000)
      .map(i -> i % 20)
      .groupBy(i->i) // no any group eviction
      .flatMap(gf -> gf.hide()
                                     .take(10)
                                     .observeOn(Schedulers.computation())
                        ,1000)
      .test()
      .awaitDone(5, TimeUnit.SECONDS)
      .assertComplete();

    Concurrency in flatMap of 1000 is not enough

  22. sobersanta commented on May 12, 2020

    @sobersanta
    ContributorAuthor

    And when group is cancelled e.g. by take() operator, and it's done asynchronously, this is again is not accounted into the pair "requested-emitted" and leads to group abandonment.
    I believe number of completed/cancelled groups should be accounted in the abandonment error generation.

  23. akarnokd commented on May 12, 2020

    @akarnokd
    Member

    Are you sure you get group abandonment instead of MissingBackpressureException? Abandonment happens only when there is no synchronous subscription happening but there were requests for new groups.

    Also if you delay the completion within flatMap, there is no way for the operator to request more until there are concurrency slots freeing up.

  24. sobersanta commented on May 12, 2020

    @sobersanta
    ContributorAuthor
  25. sobersanta commented on May 12, 2020

    @sobersanta
    ContributorAuthor
  26. akarnokd commented on May 13, 2020

    @akarnokd
    Member

    I've experimented with the new cases and here is the rundown:

    • gf.publish(shared->shared) does not hang for me.
    • gf.observeOn().take() hangs but gf.hide().observeOn().take() passes.

    The latter could lead down the rabbit hole so I'll merge #6979 to have a clean baseline for further actions. Let's continue the discussion in #6982.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions