You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Flowable.groupBy hangs with group consumed via observeOn and take(timed) #6982
The unconditional group removal in cancel(K key) could be a contributing factor; debug indicated 0 groups during the hang.
Rejected execution
take makes observeOn dispose its worker so items are stuck in the (fused) queue not being cleaned.
Currently, there is no good way to detect such rejection through the Scheduler.Worker API as rejected schedules return EmptyDisposable.INSTANCE and TrampolineScheduler also returns it as a default indicator. This would require API changes and the introduction of a dedicated and public rejected-disposable.
Second, observeOn has to be updated to detect rejection and perform the cleanup on the fused queue.
Third, the cancellation can come after the schedule but before the drain execution, thus the drain is never executed and the operator is stuck in wip > 0 state.
Difference with hide()
With hide(), the group manages the items and can request more upon cancellation, hence there should be no hanging.
when evicting groups from the map externally (ttl, size, etc) without async boundary downstream, hangs sometimes on the very last item, not every time.
Test case:
Flowable
.range(1, 500_000)
.map(i -> i % 20)
.groupBy(i -> i, i -> i, false, groupByBufferSize,
sizeAndTtlCapGuava(1000, Duration.ofMillis(10), false))
.flatMap(gf -> gf, groups)
.test()
.awaitDone(5, TimeUnit.SECONDS)
.assertComplete();
when cancelling groups e.g. by take() operator.
Here we have both issue with hangs and abandonment.
Please try
Flowable
.range(1, 500_000)
.map(i -> i % 20)
.groupBy(i->i) // no any group eviction
.flatMap(gf -> gf.take(10)
.observeOn(Schedulers.computation())
,groups)
.test()
.awaitDone(5, TimeUnit.SECONDS)
.assertComplete();
We can't do anything about the MBEs, it is inherent when running with limited concurrency because a request from one group can lead to a new group creation for which if there is no request on the main output, there is no way to store it in a bounded manner.
The objective here is to not let the chain hang but have it complete or fail with MBE depending on the situation.
This lack of coordination between group eviction/consequent flatMap's request greatly limits use cases of the operator :(
Maybe there could be considered some change in it's design so it can become possible?
E.g. queue up emissions of new groups, caused specifically by eviction (it's relatively easy to detect) until there is a request from downstream?
Or do emissions on the request, not during onNext?
The question is that there is a time gap between completion of a group and consequent request, while if it was synchronously, there wouldn't be an issue with MBE.
Issue looks exactly the same as documented requirement of a mandatory synchronous subscription to a group right after emitted.
OK Here is another thought .... Which will demonstrate that Hang issues are related to MBEs.
In the request method there is only an increment of the atomic counter.
I can't find any signs of informing upstream about the request.
The only place where request upstream happens, is during synchronous subscription - flatMap immediately prefetches upstream.
That request is a basis for all upcoming groupBy operator work.
And if there is a delay between an emission of a group and replenishing request from the downstream, there is a high possibility of a hang / MBE because noone is coordinating emissions with requests, and also noone is informing upstream about a request from downstream - if emission downstream requests do not catch up with the emissions.
Activity
akarnokd commentedon May 13, 2020
Thoughs on the cause so far:
Unconditional group removal
The unconditional group removal in
cancel(K key)could be a contributing factor; debug indicated 0 groups during the hang.Rejected execution
takemakesobserveOndispose its worker so items are stuck in the (fused) queue not being cleaned.Currently, there is no good way to detect such rejection through the
Scheduler.WorkerAPI as rejected schedules returnEmptyDisposable.INSTANCEandTrampolineScheduleralso returns it as a default indicator. This would require API changes and the introduction of a dedicated and public rejected-disposable.Second,
observeOnhas to be updated to detect rejection and perform the cleanup on the fused queue.Third, the cancellation can come after the schedule but before the drain execution, thus the drain is never executed and the operator is stuck in wip > 0 state.
Difference with hide()
With
hide(), the group manages the items and can request more upon cancellation, hence there should be no hanging.sobersanta commentedon May 13, 2020
My observations with the latest version:
Test case:
Here we have both issue with hangs and abandonment.
Please try
sobersanta commentedon May 13, 2020
akarnokd commentedon May 13, 2020
We can't do anything about the MBEs, it is inherent when running with limited concurrency because a request from one group can lead to a new group creation for which if there is no request on the main output, there is no way to store it in a bounded manner.
The objective here is to not let the chain hang but have it complete or fail with MBE depending on the situation.
sobersanta commentedon May 13, 2020
This lack of coordination between group eviction/consequent flatMap's request greatly limits use cases of the operator :(
Maybe there could be considered some change in it's design so it can become possible?
E.g. queue up emissions of new groups, caused specifically by eviction (it's relatively easy to detect) until there is a request from downstream?
Or do emissions on the request, not during onNext?
sobersanta commentedon May 13, 2020
The question is that there is a time gap between completion of a group and consequent request, while if it was synchronously, there wouldn't be an issue with MBE.
Issue looks exactly the same as documented requirement of a mandatory synchronous subscription to a group right after emitted.
sobersanta commentedon May 13, 2020
OK Here is another thought .... Which will demonstrate that Hang issues are related to MBEs.
In the request method there is only an increment of the atomic counter.
I can't find any signs of informing upstream about the request.
The only place where request upstream happens, is during synchronous subscription - flatMap immediately prefetches upstream.
That request is a basis for all upcoming groupBy operator work.
And if there is a delay between an emission of a group and replenishing request from the downstream, there is a high possibility of a hang / MBE because noone is coordinating emissions with requests, and also noone is informing upstream about a request from downstream - if emission downstream requests do not catch up with the emissions.
Makes sense?
sobersanta commentedon May 13, 2020
Please note I'm talking about requests from the flatMap, not from the consumers of the groups!
sobersanta commentedon May 18, 2020
Hi. Really don't want to bother you, but could you please let us know, what are the plans regarding groupBy operator?
sobersanta commentedon May 18, 2020
Latest snapshot still hangs for me with these tests:
First one hangs in around 50% of runs always exactly at the last item, second hangs always.
akarnokd commentedon May 18, 2020
I'll be looking into them (and a few additional cases I was thinking about the past days) tomorrow.
sobersanta commentedon May 18, 2020
Thanks!!!