Repository navigation
Coroutine with asyncio.sleep() hangs when run in a different thread #312
Description
Activity
Event loop is coupled to thread. You cannot share the loop between different threads, except
call_soon_threadsafe()/run_coroutine_threadsafe()calls.See https://docs.python.org/3/library/asyncio-dev.html#concurrency-and-multithreading
You can have a loop per thread though.
Oh right, I think I remember reading a while back that
asyncio.get_event_loop()returned the main event loop, which can only be run from the main thread. If I want to run an event loop in a child thread, do I have to create it withasyncio.new_event_loop()from the child thread, or can I create it in the main thread first and run it in the child thread?Assuming that the child thread has to create the event loop, here is updated code that creates a new event loop and runs it in the child thread. It still has the same hanging problem as before.
import asyncio from threading import Thread, Semaphore async def onRequest(data): print('got request: {}'.format(data)) await asyncio.sleep(1) print('finished processing request: {}'.format(data)) loop_created_sem = Semaphore(0) def child_executor(): global loop loop = asyncio.new_event_loop() # signal to the main thread that we've created the event loop loop_created_sem.release() loop.run_forever() child = Thread(target=child_executor, name="child") def request(data): global loop asyncio.run_coroutine_threadsafe(onRequest(data), loop) child.start() # wait for the child thread to create its event loop # before we advance loop_created_sem.acquire() request(4)
Expected Output:
got request: 4 finished processing request: 4Output (control^C pressed after 10 seconds)
got request: 4 ^CException ignored in: <module 'threading' from '/home/ubuntu/lib/python3.5/threading.py'> Traceback (most recent call last): File "/home/ubuntu/lib/python3.5/threading.py", line 1288, in _shutdown t.join() File "/home/ubuntu/lib/python3.5/threading.py", line 1054, in join self._wait_for_tstate_lock() File "/home/ubuntu/lib/python3.5/threading.py", line 1070, in _wait_for_tstate_lock elif lock.acquire(block, timeout): KeyboardInterruptThat's because you don't pass explicit event loop to
asyncio.sleep(1, loop=loop).As an option you may setup thread-local loop:
loop = asyncio.new_event_loop() asyncio.set_event_loop(loop)see also bugs.python.org/issue26969
py.test asyncio plugin has a feature that detects the frowned-upon use of the default event loop
Honestly I recommend to newer use implicit event loop in your code but always pass
loop=loopinto every asyncio API call which accepts the loop.Moreover global implicit loop should be disabled by
asyncio.set_event_loop(None)call.
This is the only way to write 100% correct code.On 18 July 2016 at 22:46, Andrew Svetlov notifications@github.com wrote:
Honestly I recommend to newer use implicit event loop in your code but
always pass loop=loop into every asyncio API call which accepts the loop.Moreover global implicit loop should be disabled by
asyncio.set_event_loop(None) call.
This is the only way to write 100% correct code.That seems an exaggeration. So if you don't add loop=loop parameters
everywhere, that code still works with the default loop. It just means
that asyncio code does not support being run in a thread, it's not the end
of the world. Just document it.Gustavo J. A. M. Carneiro
Gambit Research
"The universe is always one step beyond logic." -- Frank HerbertAs I understand it, one should set a customized event loop policy globally,
which should be able to return the right loop according to the context. For
instance, in a multithreaded context, get_event_loop() can check the thread
id and return a loop created for that thread. Passing explicitly the loop
is useful when something is expected to run on a distant loop : for
instance, one can create Task that will run in a sub-thread loop (but
loop.create_task() does this) or to passe a Future() that will be awaited
on the sub-thread loop.The thing is that the documentation is (or used to be) light on the topic,
and many people (including myself) started to write code as asyncio is
itself written : using explicit loop passing and only use get_event_loop()
as a fallback, making the the latter solution look like a sloppy
practice... and thus avoided.2016-07-19 11:44 GMT+02:00 Gustavo J. A. M. Carneiro <
notifications@github.com>:On 18 July 2016 at 22:46, Andrew Svetlov notifications@github.com wrote:
Honestly I recommend to newer use implicit event loop in your code but
always pass loop=loop into every asyncio API call which accepts the loop.Moreover global implicit loop should be disabled by
asyncio.set_event_loop(None) call.
This is the only way to write 100% correct code.That seems an exaggeration. So if you don't add loop=loop parameters
everywhere, that code still works with the default loop. It just means
that asyncio code does not support being run in a thread, it's not the end
of the world. Just document it.Gustavo J. A. M. Carneiro
Gambit Research
"The universe is always one step beyond logic." -- Frank Herbert—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
#312 (comment), or mute
the thread
https://github.com/notifications/unsubscribe-auth/AAL7GMB4ycVgXckOapVv_IneOAkww7xRks5qXJyWgaJpZM4HFw8N
.Martin http://www.martiusweb.net Richard
www.martiusweb.netI wanted to mention that in OP's code, the actual exception is caught and set in the
concurrent.Futurereturned byrun_coroutine_threadsafe. This exception is actually quite explicit about what's going on. It is possible to modify the first example to get it raised:def request(data): concurrent_future = asyncio.run_coroutine_threadsafe(onRequest(data), loop) # Raises: RuntimeError: There is no current event loop in thread 'child'. return concurrent_future.result()Regarding the proper way to handle the event loop, my recommendation for users would be to never pass it explicitly and always rely on
get_event_loop. This is perfectly compatible with threading, as long asasyncio.set_event_loopis called in the proper thread before the sub-loop is run. Using OP's example:loop = asyncio.new_event_loop() loop.set_debug(True) def child_executor(): asyncio.set_event_loop(loop) loop.run_forever()My opinion is that passing an event loop around might make sense for libraries to be unit-testable, but it is annoying and unnecessary for user code.
@vxgmichel do you mean that non-libraries should not be unit-testable?
@asvetlov My mistake, that's not what I meant.
Relying on
get_event_loopis not a problem for unit-tests: using a TestCase.new_test_loop() as the current loop works just fine:class SomeTests(test_utils.TestCase): def setUp(self): self.loop = self.new_test_loop() asyncio.set_event_loop(self.loop) def test_something(self): [...]Even
asynciodoes this kind of test in some cases. To quote Guido (see this message):I also think it's fine for a particular library or app to request that
get_event_loop() not return None. User code is simpler that way.I can only see two issues with a library not supporting explicit loop passing:
- Everything built on top of the library will have to rely on
get_event_loopas well. - Objects similar to
LockorQueuecan be created outside of the event loop so it makes sense for them to have theloop=Noneargument anyway. If the library provides such objects, separate tests will be required in order to handle this particular case.
For small libraries, those two points are probably not worth the trouble of passing the event loop around. But I guess it is also a matter of taste.
- Everything built on top of the library will have to rely on
Here is my code:
This produces the following output:
and then it just hangs, without the 'finished processing request' message. If I press ctrl^C, it outputs the following message, which doesn't look very interesting: