You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
3.0.3 - Hangs when using combination of flatMap and publish #6974
After few days of desperate attempts to find the cause of my servers' spontaneous hangs, I managed to reproduce the issue, but still don't have a clear idea about the reason, but I think I have identified a possible direction.
All problems has started when I started using very promising operator groupBy with evicting map provider possibility:
First I realised that it doesn't work well with Caffeine cache, as it performs evictions on a separate thread pool, and with async eviction notification groupBy goes crazy.
After moving to Guava things got better but only to some extent.
The tests I want to demonstrate I believe reproduce the issue I have. Definitely chain of operators is way more complicated in the production code, but here with certain variant of .publish() operator it's reproducible as well.
Basically, I'm not sure if there is issue in the groupBy(...evictingMapFactory) or in the publish(Function...), or in both.
Also tests are demonstrating that eviction policy doesn't work very well when crossing async boundaries downstream.
Please take a look at the attached file, it contains few simple tests, I kindly ask to check carefully all parameters, as it's important.
The problem is caused by the eviction double decrementing the group counter, once when the eviction completes the shared chain that cancels the group and once after the eviction.
As for the Map's remove(Object,Object).
WDYT about checking if the map is instance of ConcurrentMap (I believe it will be like that in most of the cases, I can hardly imagine anyone would uses anything else), and if yes then using ConcurrentMap.remove(key, value), otherwise keep existing logic?
That check can be done during the construction, and for example keep 2 nullable references to maps, or wrap a non-ConcurrentMap's implementation with a proper delegating wrapper.
I have tried the code from your latest PR. And my pipeline still hangs on the graceful completion :(
So far don't have a simple test which is reproducing the problem, will let you know when I get something. Could be a problem is in another operator. (Though it works fine if I serialize calls to State.cancel and upstream onNext in groupby)
I can see that partially my problems are caused by the .publish() operator (it uses FlowablePublishMulticast) which on complete calls .dispose() which in turn cancels upstream. I'm not sure I understand the reason of that canceling while this is a normal graceful shutdown. Could you please elaborate a little for a dummy like me why is it cancelling upstream on complete event? It messes with my resource cleanup logic (onTerminate is never called after that), and I'm not sure how to overrun it.
BTW here's the case where the new code fails even with .hide() in place:
With publish(Function), the inner flow may be independent of the upstream so when the inner terminates, the upstream should be stopped. I suppose when there is a causal chain from an upstream completion to a downstream completion, this cancellation could be suppressed.
Continuing about problem with the latest test above.
What I realised today is that when a group completes (e.g. because of external eviction from the Map), it pushes down onComplete event, and in case there is observeOn operator downstream, or any other reason which causes onComplete to be scheduled on another thread, the consequence is that expected group request from flatMap (or another operator consuming groups) happens with a delay, which leads to "group abandonment". It's not really about subscribing to a new group only, as stated in the documentation, but requesting more when another one completes.
Simple reproduction:
Flowable
.range(1, 500_000)
.map(i -> i % 20)
.groupBy(i->i) // no any group eviction
.flatMap(gf -> gf.hide()
.take(10)
.observeOn(Schedulers.computation())
,1000)
.test()
.awaitDone(5, TimeUnit.SECONDS)
.assertComplete();
And when group is cancelled e.g. by take() operator, and it's done asynchronously, this is again is not accounted into the pair "requested-emitted" and leads to group abandonment.
I believe number of completed/cancelled groups should be accounted in the abandonment error generation.
Are you sure you get group abandonment instead of MissingBackpressureException? Abandonment happens only when there is no synchronous subscription happening but there were requests for new groups.
Also if you delay the completion within flatMap, there is no way for the operator to request more until there are concurrency slots freeing up.
Maybe I’m calling it wrong... It’s naturally being delayed: you evict group
from the map, immediately schedule group eviction task which will in turn
call onComplete. But that will happen eventually, along with following
request from flatMap.
But in that turn of onNext a new group is already trying to be emitted
which is causing error as requests didn’t catch up with emits.
Basically - isn’t it a violation of the protocol? we shouldn’t emit anything
unless requested. I think such eviction should be coordinated with the
emission? Conceptually we have just completed previous group ( in Guava
evictions happen exactly when you call get/put) we are allowed to emit new
group, but only after it completes and flatMap requests more. So maybe it
should queue up such emissions until next request from downstream?
Activity
akarnokd commentedon May 9, 2020
Hi and thanks for reporting this bug.
The problem is caused by the eviction double decrementing the group counter, once when the eviction completes the shared chain that cancels the group and once after the eviction.
I'll post a fix shortly.
sobersanta commentedon May 9, 2020
Cool, looking forward for the fix!
akarnokd commentedon May 10, 2020
/cc @davidmoten regarding the async eviction concerns.
sobersanta commentedon May 10, 2020
Thanks for the prompt reply and the fix!
May I ask, when we can expect to have it to be released?
Maybe 3.0.4 soon? ;)
akarnokd commentedon May 10, 2020
I suggest you verify it by testing with the current RxJava snapshot, just in case: https://github.com/ReactiveX/RxJava#snapshots
sobersanta commentedon May 10, 2020
Sure, I will let you know about results.
sobersanta commentedon May 10, 2020
Could you please comment on this?
sobersanta commentedon May 10, 2020
changing
.take(10, MILLISECONDS)to.take(10)makes only 3d case failing with timeoutakarnokd commentedon May 10, 2020
Is this with the snapshot?
sobersanta commentedon May 10, 2020
yes, snapshot
12 remaining items
sobersanta commentedon May 11, 2020
More than clear, thanks for the answer!
sobersanta commentedon May 11, 2020
As for the Map's remove(Object,Object).
WDYT about checking if the map is instance of ConcurrentMap (I believe it will be like that in most of the cases, I can hardly imagine anyone would uses anything else), and if yes then using ConcurrentMap.remove(key, value), otherwise keep existing logic?
That check can be done during the construction, and for example keep 2 nullable references to maps, or wrap a non-ConcurrentMap's implementation with a proper delegating wrapper.
sobersanta commentedon May 11, 2020
I have tried the code from your latest PR. And my pipeline still hangs on the graceful completion :(
So far don't have a simple test which is reproducing the problem, will let you know when I get something. Could be a problem is in another operator. (Though it works fine if I serialize calls to State.cancel and upstream onNext in groupby)
sobersanta commentedon May 11, 2020
OK here is the case (just added
.observeOnto grouped flowable):akarnokd commentedon May 11, 2020
If you put
hide()beforeobserveOn, does it work then?sobersanta commentedon May 11, 2020
Apparently yes
sobersanta commentedon May 11, 2020
I can see that partially my problems are caused by the .publish() operator (it uses FlowablePublishMulticast) which on complete calls .dispose() which in turn cancels upstream. I'm not sure I understand the reason of that canceling while this is a normal graceful shutdown. Could you please elaborate a little for a dummy like me why is it cancelling upstream on complete event? It messes with my resource cleanup logic (onTerminate is never called after that), and I'm not sure how to overrun it.
BTW here's the case where the new code fails even with
.hide()in place:akarnokd commentedon May 11, 2020
With
publish(Function), the inner flow may be independent of the upstream so when the inner terminates, the upstream should be stopped. I suppose when there is a causal chain from an upstream completion to a downstream completion, this cancellation could be suppressed.I'll look into the new cases tomorrow.
sobersanta commentedon May 12, 2020
Adding .observeOn leads to MissingbackpressureException:
sobersanta commentedon May 12, 2020
Continuing about problem with the latest test above.
What I realised today is that when a group completes (e.g. because of external eviction from the Map), it pushes down onComplete event, and in case there is observeOn operator downstream, or any other reason which causes onComplete to be scheduled on another thread, the consequence is that expected group request from flatMap (or another operator consuming groups) happens with a delay, which leads to "group abandonment". It's not really about subscribing to a new group only, as stated in the documentation, but requesting more when another one completes.
Simple reproduction:
Concurrency in flatMap of 1000 is not enough
sobersanta commentedon May 12, 2020
And when group is cancelled e.g. by take() operator, and it's done asynchronously, this is again is not accounted into the pair "requested-emitted" and leads to group abandonment.
I believe number of completed/cancelled groups should be accounted in the abandonment error generation.
akarnokd commentedon May 12, 2020
Are you sure you get group abandonment instead of MissingBackpressureException? Abandonment happens only when there is no synchronous subscription happening but there were requests for new groups.
Also if you delay the completion within flatMap, there is no way for the operator to request more until there are concurrency slots freeing up.
sobersanta commentedon May 12, 2020
sobersanta commentedon May 12, 2020
akarnokd commentedon May 13, 2020
I've experimented with the new cases and here is the rundown:
gf.publish(shared->shared)does not hang for me.gf.observeOn().take()hangs butgf.hide().observeOn().take()passes.The latter could lead down the rabbit hole so I'll merge #6979 to have a clean baseline for further actions. Let's continue the discussion in #6982.