From 00a2487b80770fffb740dae8663459e60dea0a59 Mon Sep 17 00:00:00 2001 From: Yury Selivanov Date: Mon, 26 Oct 2020 16:55:51 -0700 Subject: [PATCH 1/8] Address feedback from Irit and Guido. --- except_star.md | 523 ++++++++++++++++++++++++++++++++----------------- 1 file changed, 345 insertions(+), 178 deletions(-) diff --git a/except_star.md b/except_star.md index 7b1dc2c..4aba174 100644 --- a/except_star.md +++ b/except_star.md @@ -1,33 +1,33 @@ -### Disclaimer +# Introducing try..except* syntax -* I'm going to be using the `ExceptionGroup` name in this issue, even - though there are other alternatives, e.g. `AggregateException`. Naming of the - "exception group" object is outside of the scope of this issue. +## Disclaimer -* This issue is primarily focused on discussing the new syntax modification - proposal for the `try..except` construct, shortly called "except*". +* We use the `ExceptionGroup` name herein, even though there + are other alternatives, e.g. `AggregateException`. Naming of the + "exception group" object is out of scope of this proposal. -* I use the term "naked" exception for regular Python exceptions - **not wrapped** in an ExceptionGroup. E.g. a regular `ValueError` +* We use the term "naked" exception for regular Python exceptions + **not wrapped** in an `ExceptionGroup`. E.g. a regular `ValueError` propagating through the stack is "naked". -* I assume that `ExceptionGroup` would be an iterable object. +* We assume that `ExceptionGroup` would be an iterable object. E.g. `list(ExceptionGroup(ValueError('a'), TypeError('b')))` would be equal to `[ValueError('a'), TypeError('b')]` -* I assume that `ExceptionGroup` won't be an indexable object; essentially +* We assume that `ExceptionGroup` won't be an indexable object; essentially it's similar to Python `set`. The motivation for this is that exceptions can occur in random order, and letting users write `group[0]` to access the - "first" error is error prone. The actual implementation of `ExceptionGroup` - will likely use an ordered list of errors though. + "first" error is error prone. Although the actual implementation of + `ExceptionGroup` will likely use an ordered list of errors to preserve + the actual occurrence order for rendering. -* I assume that `ExceptionGroup` will be a subclass of `BaseException`, +* We assume that `ExceptionGroup` will be a subclass of `BaseException`, which means it's assignable to `Exception.__context__` and can be - directly handled with `except ExceptionGroup`. + directly handled with `try: ... except ExceptionGroup: ...`. -* The behavior of good and old regular `try..except` will not be modified. +* The behavior of the regular `try..except` block will not be modified. -### Syntax +## Syntax We're considering to introduce a new variant of the `try..except` syntax to simplify working with exception groups: @@ -47,18 +47,11 @@ The new syntax can be viewed as a variant of the tuple unpacking syntax. The `*` symbol indicates that zero or more exceptions can be "caught" and processed by one `except *` clause. -We also propose to enable "unpacking" in the `raise` statement: +## Semantics -```python -errors = (ValueError('hello'), TypeError('world')) -raise *errors -``` - -### Semantics - -#### Overview +### Overview -The `except *SpamError` block will be run if the `try` code raised an +The `except *SpamError` block will be run if the `try` code raised an `ExceptionGroup` with one or more instances of `SpamError`. It would also be triggered if a naked instance of `SpamError` was raised. @@ -66,7 +59,7 @@ The `except *BazError as e` block would aggregate all instances of `BazError` into a list, wrap that list into an `ExceptionGroup` instance, and assign the resultant object to `e`. The type of `e` would be `ExceptionGroup[BazError]`. If there was just one naked instance of -`BazError`, it would be wrapped into a list and assigned to `e`. +`BazError`, it would be wrapped into an `ExceptionGroup` and assigned to `e`. The `except *(BarError, FooError) as e` would aggregate all instances of `BarError` or `FooError` into a list and assign that wrapped list to `e`. @@ -91,7 +84,7 @@ except *CancelledError: pass ``` -Exceptions are mached using a subclass check. For example: +Exceptions are matched using a subclass check. For example: ```python try: @@ -111,10 +104,9 @@ InterruptedError BlockingIOError ``` -#### New raise* Syntax +### Raising ExceptionGroups manually -The new `raise *` syntax allows to users to only process some exceptions -out of the matched set, e.g.: +Exception groups can be raised manually: ```python try: @@ -124,103 +116,161 @@ except *OSerror as errors: for e in errors: if e.errno != errno.EPIPE: new_errors.append(e) - raise *new_errors + raise ExceptionGroup(*new_errors) ``` -The above code ignores all `EPIPE` OS errors, while letting all others -propagate. +The above code ignores all `EPIPE` OS errors, while letting all other +exceptions propagate. -`raise *` syntax is special: it effectively extends the exception group with -a list of errors without creating a new `ExceptionGroup` instance: +Raising an `ExceptionGroup` introduces nesting: ```python try: - raise *(ValueError('a'), TypeError('b')) + raise ExceptionGroup(ValueError('a'), TypeError('b')) except *ValueError: - raise *(KeyError('x'), KeyError('y')) + raise ExceptionGroup(KeyError('x'), KeyError('y')) # would result in: -# ExceptionGroup({KeyError('x'), KeyError('y'), TypeError('b')}) +# +# ExceptionGroup( +# ExceptionGroup( +# KeyError('x'), +# KeyError('y'), +# ), +# TypeError('b'), +# ) ``` -A regular raise would behave similarly: +Although a regular `raise Exception` would not wrap `Exception` in a group: ```python try: - raise *(ValueError('a'), TypeError('b')) + raise ExceptionGroup(ValueError('a'), TypeError('b')) except *ValueError: raise KeyError('x') # would result in: -# ExceptionGroup({KeyError('x'), TypeError('b')}) +# +# ExceptionGroup( +# KeyError('x'), +# TypeError('b') +# ) ``` -`raise *` accepts arguments of type `Iterable[BaseException]`. - -#### Unmatched Exceptions +### Unmatched Exceptions Example: ```python try: - raise *(ValueError('a'), TypeError('b'), TypeError('c'), KeyError('e')) + raise ExceptionGroup( + ValueError('a'), TypeError('b'), TypeError('c'), KeyError('e') + ) except *ValueError as e: print(f'got some ValueErrors: {e}') except *TypeError as e: print(f'got some TypeErrors: {e}') - raise *e + raise e ``` The above code would print: ``` -got some ValueErrors: ExceptionGroup({ValueError('a')}) -got some TypeErrors: ExceptionGroup({TypeError('b'), TypeError('c')}) +got some ValueErrors: ExceptionGroup(ValueError('a')) +got some TypeErrors: ExceptionGroup(TypeError('b'), TypeError('c')) ``` -And then crash with an unhandled `KeyError('e')` error. +and then crash with an unhandled `ExceptionGroup`: + +``` +ExceptionGroup( + KeyError('e'), + ExceptionGroup( + TypeError('b'), + TypeError('c') + ) +) +``` Basically, before interpreting `except *` clauses, the interpreter will -have an exception group object with a list of exceptions in it. Every -`except *` clause, evaluated from top to bottom, can filter some of the -exceptions out of the group and process them. In the end, if the exception -group has no exceptions left in it, it wold mean that all exceptions were -processed. If the exception group has some unprocessed exceptions, the current -frame will be "pushed" to the group's traceback and the group would be -propagated up the stack. +have an "incoming" `ExceptionGroup` object with a list of exceptions in it +to handle, and then: + +* A new empty "result" `ExceptionGroup` would be created by the interpreter. + +* Every `except *` clause, run from top to bottom, can filter some of the + exceptions out of the group and process them. If the except block crashes + with an error, that error is put to the "result" `ExceptionGroup` (with the + group of unprocessed exceptions referenced via the `__context__` attribute.) + +* After there are no more `except*` clauses to evaluate, there are the + following possibilities: + + * Both "incoming" and "result" `ExceptionGroup` are empty. This means + that all exceptions were processed and silenced successfully. + + * Both "incoming" and "result" `ExceptionGroup` are not empty. + This means that not all of the exceptions were matched, and some were + matched but either triggered new errors, or were re-raised. The interpreter + would merge both groups into one group and raise it. + + * The "incoming" `ExceptionGroup` is non-empty: not all exceptions were + processed. The interpreter would raise the "incoming" group. + + * The "result" `ExceptionGroup` is non-empty: all exceptions were processed, + but some were re-raised or caused new errors. The interpreter would + raise the "result" group. + +The order of `except*` clauses is significant just like with the regular +`try..except`, e.g.: + +```python +try: + raise BlockingIOError +except *OSError as e: + # Would catch the error + print(e) +except *BlockingIOError: + # Would never run + print('never') + +# would print: +# +# ExceptionGroup(BlockingIOError()) +``` -#### Exception Chaining +### Exception Chaining -If an error occur during processing a set of exceptions in a `except *` block, -all matched errors would be put in a new `ExceptionGroup` which would have its -`__context__` attribute set to the just occurred exception: +If an error occurs during processing a set of exceptions in a `except *` block, +all matched errors would be put in a new `ExceptionGroup` which would be +references from the just occurred exception via its `__context__` attribute: ```python try: - raise *(ValueError('a'), ValueError('b'), TypeError('z')) + raise ExceptionGroup(ValueError('a'), ValueError('b'), TypeError('z')) except *ValueError: 1 / 0 # would result in: # -# ExceptionGroup({ +# ExceptionGroup( # TypeError('z'), # ZeroDivisionError() -# }) +# ) # # where the `ZeroDivizionError()` instance would have # its __context__ attribute set to # -# ExceptionGroup({ +# ExceptionGroup( # ValueError('a'), ValueError('b') -# }) +# ) ``` It's also possible to explicitly chain exceptions: ```python try: - raise *(ValueError('a'), ValueError('b'), TypeError('z')) + raise ExceptionGroup(ValueError('a'), ValueError('b'), TypeError('z')) except *ValueError as errors: raise RuntimeError('unexpected values') from errors @@ -234,20 +284,177 @@ except *ValueError as errors: # where the `RuntimeError()` instance would have # its __cause__ attribute set to # -# ExceptionGroup({ -# ValueError('a'), ValueError('b') -# }) +# ExceptionGroup( +# ValueError('a'), +# ValueError('b') +# ) +``` + +### Recursive Matching + +The matching of `except *` clauses against an `ExceptionGroup` is performed +recursively. E.g.: + +```python +try: + raise ExceptionGroup( + ValueError('a'), + TypeError('b'), + ExceptionGroup( + TypeError('c'), + KeyError('d') + ) + ) +except *TypeError as e: + print(f'got some TypeErrors: {len(e)}') +except *Exception: + pass + +# would print: +# +# got some TypeErrors: 2 +``` + +Iteration over an `ExceptionGroup` that has nested `ExceptionGroup` objects +in it effectively flattens the entire tree. E.g. + +```python +print( + list( + ExceptionGroup( + ValueError('a'), + TypeError('b'), + ExceptionGroup( + TypeError('c'), + KeyError('d') + ) + ) + ) +) + +# would output: +# +# [ValueError('a'), TypeError('b'), TypeError('c'), KeyError('d')] +``` + +### "raise e" vs "raise" + +The difference between bare `raise` and a more specific `raise e` is more +significant for exception groups than it is for regular exceptions. Consider +the following two examples that illustrate it. + +Bare `raise` preserves the exception group internal structure: + +```python +try: + raise ExceptionGroup( + ValueError('a'), + TypeError('b'), + ExceptionGroup( + TypeError('c'), + KeyError('d') + ) + ) +except *TypeError as e: + raise + +# would crash with: +# +# ExceptionGroup( +# ValueError('a'), +# TypeError('b'), +# ExceptionGroup( +# TypeError('c'), +# KeyError('d') +# ) +# ) +``` + +Whereas `raise e` would flatten the captured subset: + +```python +try: + raise ExceptionGroup( + ValueError('a'), + TypeError('b'), + ExceptionGroup( + TypeError('c'), + KeyError('d') + ) + ) +except *TypeError as e: + raise e + +# would crash with: +# +# ExceptionGroup( +# ValueError('a'), +# ExceptionGroup( +# TypeError('b'), +# TypeError('c'), +# ), +# ExceptionGroup( +# KeyError('d') +# ) +# ) ``` -### Types of errors +### "continue" and "break" in "except*" + +Both `continue` and `break` are disallowed in `except*` clauses, causing +a `SyntaxError`. + +Due to the fact that `try..except*` block allows multiple `except*` clauses +to run while handling one `ExceptionGroup` with multiple different exceptions +in it, allowing one innocent `break` or `continue` in one `except*` to +effectively silence the entire group feels very error prone. + +### "return" in "except*" + +A `return` in a regular `except` or `finally` clause means +"suppress the exception". For example, both of the below functions would +silence their `ZeroDivisionError`s: + +```python +def foo(): + try: + 1 / 0 + finally: + print('the sound of') + return + +def bar(): + try: + 1 / 0 + except ZeroDivisionError: + pass + finally: + print('silence') + return + +foo() +bar() + +# would print: +# +# the sound of +# silence +``` + +We propose to replicate this behavior in the `except*` syntax as it is useful +as an escape hatch when it's clear that all exceptions can be silenced. + +## Design Considerations + +### Why try..except* syntax Fundamentally there are two kinds of exceptions: *control flow exceptions* -and *operation exceptions*. The examples of former are `KeyboardInterrupt`, -`asyncio.CancelledError`, etc. Latter are `TypeError`, `KeyError`, etc. +(e.g. `KeyboardInterrupt` or `asyncio.CancelledError`) and +*operation exceptions* (e.g. `TypeError` or `KeyError`). When writing async/await code that uses a concept of TaskGroups (or Trio's nurseries) to schedule different code concurrently, the users should -approach these kinds in a fundamentally different way. +approach these two kinds in a fundamentally different way. *Operation exceptions* such as `KeyError` should be handled within the async Task that runs the code. E.g. this is what users should do: @@ -270,22 +477,27 @@ except *KeyError: # no context to do anything with it but to log it. ``` -*Control flow exceptions* are a different beast. If, for example, we want to -cancel an asyncio Task that spawned multiple concurrent Tasks in it with a -TaskGroup, we need to make sure that: +*Control flow exceptions* are different. If, for example, we want to +cancel an asyncio Task that spawned other multiple concurrent Tasks in it +with a an `asyncio.TaskGroup`, the following will happen: * CancelledErrors will be propagated to all still running tasks within - the group + the group; -* CancelledErrors will be propagated to the Task that scheduled the group +* CancelledErrors will be propagated to the Task that scheduled the group and + bubble up from `async with TaskGroup()`; -* CancelledErrors will be propagated to the outer Tasks until either the entire - program stops, or the exception is handled and silenced - (e.g. by `asyncio.wait_for()`). +* CancelledErrors will be propagated to the outer Task until either the entire + program shuts down with a `CancelledError`, or the cancellation is handled + and silenced (e.g. by `asyncio.wait_for()`). -So suppose we have the `except *ExceptionType` syntax that can only handle -an ExceptionGroup with ExceptionType in it. This means that we'd see a lot -of code like this: +*Control flow exceptions* alter the execution flow of a program. +Therefore it is sometimes desirable for the user to react to them and +run code, for example, to free resources. + +Suppose we have the `except *ExceptionType` syntax that only matches +`ExceptionGroup[ExceptionType]` exceptions (a naked `ExceptionType` wouldn't +be matched). This means that we'd see a lot of code duplication: ```python @@ -303,31 +515,31 @@ except CancelledError: raise ``` -Which led me to the conclusion that `except *CancelledError as e` should both: +Which leads to the conclusion that `except *CancelledError as e` should both: -* catch an individual standalone `CancelledError`, wrap it in an ExceptionGroup - and bind it to `e`. The type of `e` is always - `ExceptionGroup[CancelledError]`. +* catch a naked `CancelledError`, wrap it in an `ExceptionGroup` and bind it + to `e`. The type of `e` would always be `ExceptionGroup[CancelledError]`. * if an exception group is propagating through the `try`, `except *CancelledError` should split the group and handle all exceptions - at once with one run of the code in `except *CancelledError`. - -Why "handle all exceptions at once with one run of the code code in except *"? -Why not run the code in the `except` clause for every matching exception that -we have in the group? Basically because there's no need to. As I mentioned -above, catching *operation exceptions* should be done with the regular old -`except KeyError` within the Task boundary, where there's context to handle -that `KeyError`. Catching *control flow exceptions* is needed to **react** -to some global signal, do cleanup or logging, but ultimately to either -**stop** the signal **or propagate** it up the caller chain. - -Separating exceptions kinds to two distinct groups (operation & control flow) + at once with one run of the code in `except *CancelledError` (and not + run the code for every matched individual exception.) + +Why "handle all exceptions at once"? Why not run the code in the `except` +clause for every matched exception that we have in the group? +Basically because there's no need to. As we mentioned above, catching +*operation exceptions* should be done with the regular `except KeyError` +within the Task boundary, where there's context to handle a `KeyError`. +Catching *control flow exceptions* is needed to **react** to a global +signal, do cleanup or logging, but ultimately to either **stop** the signal +**or propagate** it up the caller chain. + +Separating exception kinds to two distinct groups (operation & control flow) leads to another conclusion: an individual `try..except` block usually handles -either the former or the latter, **but not a mix of both**. Which led me to +either the former or the latter, **but not a mix of both**. Which leads to the conclusion that `except *CancelledError` should switch the behavior of the entire `try` block to make it run several of its `except *` clauses if -necessary. So: +necessary. Therefore: ```python try: @@ -338,7 +550,8 @@ except ValueError: # handle ``` -is the old and familiar `try..except`, we don't need to change it. And: +is a regular `try..except` block to be used for reacting to +*operation exceptions*. And: ```python try: @@ -349,99 +562,53 @@ except *CancelledError: # handle ``` -is an entirely different mode and it's **OK**, and moreover, almost expected -from the user standpoint, to run both `except` clauses here. +is an entirely different construct meant to make it easier to react to +*control flow* signals. When specified that way, it is expected from the user +standpoint that both `except` clauses can be potentially run. -And: +Lastly, code that combines handling of both operation and control flow +exceptions is unrealistic and impractical, e.g.: ```python try: - # code + async with TaskGroup() as g: + g.create_task(task1()) + g.create_task(task2()) except ValueError: - # handle + # handle ValueError except *CancelledError: - # handle + # handle cancellation + raise ``` -is weird and most likely suggests that the code should be refactored. +In the above snippet it is impossible to attribute which task raised a +`ValueError` -- `task1` or `task2`. So it really should be handled directly +in those tasks. Whereas handling `*CancelledError` makes sense -- it means that +the current task is being canceled and this might be a good opportunity to do +a cleanup. + -### Types of user code +### Adoption of try..except* syntax -Fundamentally we have applications and libraries. Applications are somewhat -simpler -- they typically can dictate what version of Python they require. +Application code typically can dictates what version of Python it requires. Which makes introducing TaskGroups and the new `except *` clause somewhat -easier. Basically, upon switching to Python 3.10, the application developer +straightforward. Upon switching to Python 3.10, the application developer can grep their application code for every *control flow* exception they handle (search for `except CancelledError`) and mechanically change it to -`except *CancelledError`. Generally, judging by my own experience, there are -not so many places with code like that. Typically things like `CancelledError` -and `TimeoutError` are handled only in a few places, the rest of the code -relies on `try..finally` to cleanup its state. - -Library developers are in a worse position: they'll need to maintain backwards -compatibility with older Pythons, so they can't start using the new `except *` -syntax. Two thoughts here: - -This means that we'll need to have a proper programmatic API to work with -ExceptionGroups, so that libraries can write code like: - -```python -try: - # run user code -except CancelledError: - # handle cancellation -except ExceptionGroup as e: - g1, g2 = e.split(CancelledError) - # handle cancellation -``` - -The API isn't going to be pretty, but that's OK, because a lot of existing -asyncio libraries don't run user-provided coroutines that might use TaskGroups. -In other words, a mysql or redis driver will never be able to catch an -ExceptionGroup until it starts using TaskGroups itself. - -### Summary - -I just cannot wrap my head around introducing `try..catch` syntax to Python. -I don't think we actually need it, with the right approach to documentation we -can explain to the users how to use TaskGroups and `except *` syntax correctly -and I believe it will only make their code better. Therefore, -to conclude I think we should: - -* Introduce the ExceptionGroup object. Tweak the interpreter to recognize them -and do the correct thing to the wrapped Tracebacks etc. +`except *CancelledError`. -* ExceptionGroups should have an API so that the few libraries that run user -code can maintain compatibility with older Pythons. +Library developers, on the other hand, will need to maintain backwards +compatibility with older Python versions, and therefore they wouldn't be able +to start using the new `except *` syntax right away. They will have to use +the new ExceptionGroup low-level APIs along with `try..except ExceptionGroup` +to support running user code that can raise exception groups. -* `except *CancelledError as e` should catch: - * an individual `CancelledError`; it would be wrapped in an ExceptionGroup - created ad-hoc. - * a single `CancelledError` wrapped in an ExceptionGroup - * multiple `CancelledError`s at once. - -* If a `try` block has two or more `except *` clauses it can run them one after - another. - -* Perhaps, if there's one `except *` in a `try` block, we should **require** - all other `except` clauses to be written as `except *`. - -* Handling exceptions like `*TimeoutError` is rather pointless. It's hard - to correlate what specific subset of child tasks timed out -- the user - should really handle timeouts in the sub-tasks where you have the context. - So IMO exceptions groups are all about handling global signals like - CancelledError or KeyboardInterrupt and about preserving the clear history - of what crashed where and how to simplify development, debugging, - and diagnostics. - -### See Also +## See Also * An analysis of how exception groups will likely be used in asyncio programs: https://github.com/python/exceptiongroups/issues/3#issuecomment-716203284 -* A WIP implementation of the `ExceptionGroup` type by @iritkatriel +* A WIP implementation of the `ExceptionGroup` type by @iritkatriel tracked [here](https://github.com/iritkatriel/cpython/tree/exceptionGroup). - - From c04f7f3611a6a2d2a4ede2785a917a2d24a996dd Mon Sep 17 00:00:00 2001 From: Yury Selivanov Date: Mon, 26 Oct 2020 17:58:41 -0700 Subject: [PATCH 2/8] Address feedback from Guido --- except_star.md | 69 +++++++++++++++++++++++++++++++++++--------------- 1 file changed, 49 insertions(+), 20 deletions(-) diff --git a/except_star.md b/except_star.md index 4aba174..e4b18fb 100644 --- a/except_star.md +++ b/except_star.md @@ -2,30 +2,30 @@ ## Disclaimer -* We use the `ExceptionGroup` name herein, even though there - are other alternatives, e.g. `AggregateException`. Naming of the - "exception group" object is out of scope of this proposal. +* We use the `ExceptionGroup` name, even though there + are other alternatives, e.g. `AggregateException` and `MultiError`. + Naming of the "exception group" object is out of scope of this proposal. * We use the term "naked" exception for regular Python exceptions **not wrapped** in an `ExceptionGroup`. E.g. a regular `ValueError` propagating through the stack is "naked". -* We assume that `ExceptionGroup` would be an iterable object. - E.g. `list(ExceptionGroup(ValueError('a'), TypeError('b')))` would be +* `ExceptionGroup` is an iterable object. + E.g. `list(ExceptionGroup(ValueError('a'), TypeError('b')))` is equal to `[ValueError('a'), TypeError('b')]` -* We assume that `ExceptionGroup` won't be an indexable object; essentially +* `ExceptionGroup` is not an indexable object; essentially it's similar to Python `set`. The motivation for this is that exceptions can occur in random order, and letting users write `group[0]` to access the "first" error is error prone. Although the actual implementation of `ExceptionGroup` will likely use an ordered list of errors to preserve the actual occurrence order for rendering. -* We assume that `ExceptionGroup` will be a subclass of `BaseException`, - which means it's assignable to `Exception.__context__` and can be +* `ExceptionGroup` is a subclass of `BaseException`, + is assignable to `Exception.__context__`, and can be directly handled with `try: ... except ExceptionGroup: ...`. -* The behavior of the regular `try..except` block will not be modified. +* The behavior of the regular `try..except` statement will not be modified. ## Syntax @@ -71,7 +71,7 @@ example, both `except *SpamError:` and `except *(BarError, FooError) as e:` could get executed during handling of one `ExceptionGroup` object, or all of the `except*` clauses, or just one of them. -It is not allowed to use both regular `except` clauses and the new `except*` +It is not allowed to use both regular except blocks and the new `except*` clauses in the same `try` block. E.g. the following example would raise a `SyntaxErorr`: @@ -170,7 +170,7 @@ except *ValueError as e: print(f'got some ValueErrors: {e}') except *TypeError as e: print(f'got some TypeErrors: {e}') - raise e + raise ``` The above code would print: @@ -180,7 +180,7 @@ got some ValueErrors: ExceptionGroup(ValueError('a')) got some TypeErrors: ExceptionGroup(TypeError('b'), TypeError('c')) ``` -and then crash with an unhandled `ExceptionGroup`: +and then terminate with an unhandled `ExceptionGroup`: ``` ExceptionGroup( @@ -199,7 +199,7 @@ to handle, and then: * A new empty "result" `ExceptionGroup` would be created by the interpreter. * Every `except *` clause, run from top to bottom, can filter some of the - exceptions out of the group and process them. If the except block crashes + exceptions out of the group and process them. If the except block terminates with an error, that error is put to the "result" `ExceptionGroup` (with the group of unprocessed exceptions referenced via the `__context__` attribute.) @@ -207,7 +207,7 @@ to handle, and then: following possibilities: * Both "incoming" and "result" `ExceptionGroup` are empty. This means - that all exceptions were processed and silenced successfully. + that all exceptions were processed and silenced. * Both "incoming" and "result" `ExceptionGroup` are not empty. This means that not all of the exceptions were matched, and some were @@ -258,7 +258,7 @@ except *ValueError: # ZeroDivisionError() # ) # -# where the `ZeroDivizionError()` instance would have +# where the `ZeroDivisionError()` instance would have # its __context__ attribute set to # # ExceptionGroup( @@ -358,7 +358,7 @@ try: except *TypeError as e: raise -# would crash with: +# would terminate with: # # ExceptionGroup( # ValueError('a'), @@ -385,7 +385,7 @@ try: except *TypeError as e: raise e -# would crash with: +# would terminate with: # # ExceptionGroup( # ValueError('a'), @@ -427,10 +427,9 @@ def bar(): try: 1 / 0 except ZeroDivisionError: - pass + return finally: print('silence') - return foo() bar() @@ -444,6 +443,36 @@ bar() We propose to replicate this behavior in the `except*` syntax as it is useful as an escape hatch when it's clear that all exceptions can be silenced. +That said, the regular try statement allows to return a value from the except +or the finally clause: + +```python +def bar(): + try: + 1 / 0 + except ZeroDivisionError: + return 42 + +print(bar()) + +# would print "42" +``` + +Allowing non-None returns in `except*` allows to write unpredictable code, +e.g.: + +```python +try: + raise ExceptionGroup(A(), B()) +except *A: + return 1 +except *B: + return 2 +``` + +Therefore non-None returns are disallowed in `except*` clauses. + + ## Design Considerations ### Why try..except* syntax @@ -525,7 +554,7 @@ Which leads to the conclusion that `except *CancelledError as e` should both: at once with one run of the code in `except *CancelledError` (and not run the code for every matched individual exception.) -Why "handle all exceptions at once"? Why not run the code in the `except` +Why "handle all exceptions at once"? Why not run the code in the except clause for every matched exception that we have in the group? Basically because there's no need to. As we mentioned above, catching *operation exceptions* should be done with the regular `except KeyError` From f5da6d125797841d16b4a7bee7af9f33744ab17c Mon Sep 17 00:00:00 2001 From: Yury Selivanov Date: Mon, 26 Oct 2020 18:50:15 -0700 Subject: [PATCH 3/8] Prohibit returning from except* --- except_star.md | 81 ++++++++++++-------------------------------------- 1 file changed, 19 insertions(+), 62 deletions(-) diff --git a/except_star.md b/except_star.md index e4b18fb..9c142f2 100644 --- a/except_star.md +++ b/except_star.md @@ -399,79 +399,36 @@ except *TypeError as e: # ) ``` -### "continue" and "break" in "except*" +### "continue", "break", and "return" in "except*" -Both `continue` and `break` are disallowed in `except*` clauses, causing -a `SyntaxError`. +`continue`, `break`, and `return` are disallowed in `except*` clauses, +causing a `SyntaxError`. -Due to the fact that `try..except*` block allows multiple `except*` clauses -to run while handling one `ExceptionGroup` with multiple different exceptions -in it, allowing one innocent `break` or `continue` in one `except*` to -effectively silence the entire group feels very error prone. - -### "return" in "except*" - -A `return` in a regular `except` or `finally` clause means -"suppress the exception". For example, both of the below functions would -silence their `ZeroDivisionError`s: +Consider if they were allowed: ```python def foo(): - try: - 1 / 0 - finally: - print('the sound of') - return - -def bar(): - try: - 1 / 0 - except ZeroDivisionError: - return - finally: - print('silence') - -foo() -bar() - -# would print: -# -# the sound of -# silence -``` - -We propose to replicate this behavior in the `except*` syntax as it is useful -as an escape hatch when it's clear that all exceptions can be silenced. - -That said, the regular try statement allows to return a value from the except -or the finally clause: - -```python -def bar(): - try: - 1 / 0 - except ZeroDivisionError: - return 42 - -print(bar()) - -# would print "42" -``` - -Allowing non-None returns in `except*` allows to write unpredictable code, -e.g.: - -```python -try: + try: raise ExceptionGroup(A(), B()) -except *A: + except *A: return 1 -except *B: + except *B: return 2 + +print(foo()) ``` -Therefore non-None returns are disallowed in `except*` clauses. +In the above example the user could guess that most likely the program +would print "1". But if instead of a simple `raise ExceptionGroup(A(), B())` +there's scheduling of a few concurrent tasks the answer is no longer obvious. + +Ultimately though, due to the fact that `try..except*` block allows multiple +`except*` clauses to run while handling one `ExceptionGroup` with +multiple different exceptions in it, allowing one innocent `break`, `continue`, +or `return` in one `except*` to effectively silence the entire group of +errors is error prone. +We can consider allowing some of them in future versions of Python. ## Design Considerations From 90872d2b6689bd867883e4baac3b37338c7c6411 Mon Sep 17 00:00:00 2001 From: Yury Selivanov Date: Mon, 26 Oct 2020 18:56:07 -0700 Subject: [PATCH 4/8] Erase difference between `raise` and `raise e` --- except_star.md | 18 +++++++++--------- 1 file changed, 9 insertions(+), 9 deletions(-) diff --git a/except_star.md b/except_star.md index 9c142f2..6d3beb4 100644 --- a/except_star.md +++ b/except_star.md @@ -337,13 +337,9 @@ print( # [ValueError('a'), TypeError('b'), TypeError('c'), KeyError('d')] ``` -### "raise e" vs "raise" +### "raise e" and "raise" -The difference between bare `raise` and a more specific `raise e` is more -significant for exception groups than it is for regular exceptions. Consider -the following two examples that illustrate it. - -Bare `raise` preserves the exception group internal structure: +There is no difference between bare `raise` and a more specific `raise e`: ```python try: @@ -356,7 +352,7 @@ try: ) ) except *TypeError as e: - raise + raise # or "raise e" # would terminate with: # @@ -370,7 +366,11 @@ except *TypeError as e: # ) ``` -Whereas `raise e` would flatten the captured subset: +It is important to point out that the `ExceptionGroup` bound to `e` is an +ephemeral object. Raising it via `raise` or `raise e` will not cause changes +to the overall shape of the `ExceptionGroup`. If the user wants to "flatten" +the tree, they can explicitly create a new `ExceptionGroup` and raise it: + ```python try: @@ -383,7 +383,7 @@ try: ) ) except *TypeError as e: - raise e + raise ExceptionGroup(*e) # would terminate with: # From 9cbc71a41b970e5f055f5a7218008d2c312318da Mon Sep 17 00:00:00 2001 From: Irit Katriel Date: Tue, 27 Oct 2020 11:08:05 +0000 Subject: [PATCH 5/8] Minor edits --- except_star.md | 25 ++++++++++++++----------- 1 file changed, 14 insertions(+), 11 deletions(-) diff --git a/except_star.md b/except_star.md index 6d3beb4..fe23ae9 100644 --- a/except_star.md +++ b/except_star.md @@ -29,7 +29,7 @@ ## Syntax -We're considering to introduce a new variant of the `try..except` syntax to +We're proposing to introduce a new variant of the `try..except` syntax to simplify working with exception groups: ```python @@ -65,11 +65,13 @@ The `except *(BarError, FooError) as e` would aggregate all instances of `BarError` or `FooError` into a list and assign that wrapped list to `e`. The type of `e` would be `ExceptionGroup[Union[BarError, FooError]]`. -Even though every `except*` star can be called only once, any number of +Even though every `except*` clause can be executed only once, any number of them can be run during handling of an `ExceptionGroup`. E.g. in the above example, both `except *SpamError:` and `except *(BarError, FooError) as e:` could get executed during handling of one `ExceptionGroup` object, or all -of the `except*` clauses, or just one of them. +of the `except*` clauses, or just one of them. However, each exception in +the exception group is only handled by one except* clause -- the first one +that matches its type. It is not allowed to use both regular except blocks and the new `except*` clauses in the same `try` block. E.g. the following example would raise a @@ -199,17 +201,17 @@ to handle, and then: * A new empty "result" `ExceptionGroup` would be created by the interpreter. * Every `except *` clause, run from top to bottom, can filter some of the - exceptions out of the group and process them. If the except block terminates - with an error, that error is put to the "result" `ExceptionGroup` (with the + exceptions out of the group and process them. If the except block raises an + exception, that exception is added to the "result" `ExceptionGroup` (with the group of unprocessed exceptions referenced via the `__context__` attribute.) * After there are no more `except*` clauses to evaluate, there are the following possibilities: - * Both "incoming" and "result" `ExceptionGroup` are empty. This means + * Both "incoming" and "result" `ExceptionGroup`s are empty. This means that all exceptions were processed and silenced. - * Both "incoming" and "result" `ExceptionGroup` are not empty. + * Both "incoming" and "result" `ExceptionGroup`s are not empty. This means that not all of the exceptions were matched, and some were matched but either triggered new errors, or were re-raised. The interpreter would merge both groups into one group and raise it. @@ -243,7 +245,7 @@ except *BlockingIOError: If an error occurs during processing a set of exceptions in a `except *` block, all matched errors would be put in a new `ExceptionGroup` which would be -references from the just occurred exception via its `__context__` attribute: +referenced from the just occurred exception via its `__context__` attribute: ```python try: @@ -262,7 +264,8 @@ except *ValueError: # its __context__ attribute set to # # ExceptionGroup( -# ValueError('a'), ValueError('b') +# ValueError('a'), +# ValueError('b') # ) ``` @@ -422,7 +425,7 @@ In the above example the user could guess that most likely the program would print "1". But if instead of a simple `raise ExceptionGroup(A(), B())` there's scheduling of a few concurrent tasks the answer is no longer obvious. -Ultimately though, due to the fact that `try..except*` block allows multiple +Ultimately though, due to the fact that a `try..except*` block allows multiple `except*` clauses to run while handling one `ExceptionGroup` with multiple different exceptions in it, allowing one innocent `break`, `continue`, or `return` in one `except*` to effectively silence the entire group of @@ -576,7 +579,7 @@ a cleanup. ### Adoption of try..except* syntax -Application code typically can dictates what version of Python it requires. +Application code typically can dictate what version of Python it requires. Which makes introducing TaskGroups and the new `except *` clause somewhat straightforward. Upon switching to Python 3.10, the application developer can grep their application code for every *control flow* exception they handle From dea774191b5500fd5d3d1be8be68696f9842dd00 Mon Sep 17 00:00:00 2001 From: Yury Selivanov Date: Tue, 27 Oct 2020 09:02:25 -0700 Subject: [PATCH 6/8] Move the "Unmatched exceptions" section up, as @iritkatriel suggested --- except_star.md | 112 ++++++++++++++++++++++++------------------------- 1 file changed, 55 insertions(+), 57 deletions(-) diff --git a/except_star.md b/except_star.md index fe23ae9..30e5779 100644 --- a/except_star.md +++ b/except_star.md @@ -106,59 +106,6 @@ InterruptedError BlockingIOError ``` -### Raising ExceptionGroups manually - -Exception groups can be raised manually: - -```python -try: - low_level_os_operation() -except *OSerror as errors: - new_errors = [] - for e in errors: - if e.errno != errno.EPIPE: - new_errors.append(e) - raise ExceptionGroup(*new_errors) -``` - -The above code ignores all `EPIPE` OS errors, while letting all other -exceptions propagate. - -Raising an `ExceptionGroup` introduces nesting: - -```python -try: - raise ExceptionGroup(ValueError('a'), TypeError('b')) -except *ValueError: - raise ExceptionGroup(KeyError('x'), KeyError('y')) - -# would result in: -# -# ExceptionGroup( -# ExceptionGroup( -# KeyError('x'), -# KeyError('y'), -# ), -# TypeError('b'), -# ) -``` - -Although a regular `raise Exception` would not wrap `Exception` in a group: - -```python -try: - raise ExceptionGroup(ValueError('a'), TypeError('b')) -except *ValueError: - raise KeyError('x') - -# would result in: -# -# ExceptionGroup( -# KeyError('x'), -# TypeError('b') -# ) -``` - ### Unmatched Exceptions Example: @@ -186,11 +133,9 @@ and then terminate with an unhandled `ExceptionGroup`: ``` ExceptionGroup( + TypeError('b'), + TypeError('c'), KeyError('e'), - ExceptionGroup( - TypeError('b'), - TypeError('c') - ) ) ``` @@ -241,6 +186,59 @@ except *BlockingIOError: # ExceptionGroup(BlockingIOError()) ``` +### Raising ExceptionGroups manually + +Exception groups can be raised manually: + +```python +try: + low_level_os_operation() +except *OSerror as errors: + new_errors = [] + for e in errors: + if e.errno != errno.EPIPE: + new_errors.append(e) + raise ExceptionGroup(*new_errors) +``` + +The above code ignores all `EPIPE` OS errors, while letting all other +exceptions propagate. + +Raising an `ExceptionGroup` introduces nesting: + +```python +try: + raise ExceptionGroup(ValueError('a'), TypeError('b')) +except *ValueError: + raise ExceptionGroup(KeyError('x'), KeyError('y')) + +# would result in: +# +# ExceptionGroup( +# ExceptionGroup( +# KeyError('x'), +# KeyError('y'), +# ), +# TypeError('b'), +# ) +``` + +Although a regular `raise Exception` would not wrap `Exception` in a group: + +```python +try: + raise ExceptionGroup(ValueError('a'), TypeError('b')) +except *ValueError: + raise KeyError('x') + +# would result in: +# +# ExceptionGroup( +# KeyError('x'), +# TypeError('b') +# ) +``` + ### Exception Chaining If an error occurs during processing a set of exceptions in a `except *` block, From 610b4dbed79df4d6b7edc920263f2c5b19670e44 Mon Sep 17 00:00:00 2001 From: Yury Selivanov Date: Tue, 27 Oct 2020 09:27:12 -0700 Subject: [PATCH 7/8] Clarify the difference in tracebacks between `raise` and `raise e` --- except_star.md | 81 ++++++++++++++++++++++++++++++++++++++++---------- 1 file changed, 66 insertions(+), 15 deletions(-) diff --git a/except_star.md b/except_star.md index 30e5779..bad82d1 100644 --- a/except_star.md +++ b/except_star.md @@ -307,13 +307,13 @@ try: ) ) except *TypeError as e: - print(f'got some TypeErrors: {len(e)}') + print(f'got some TypeErrors: {list(e)}') except *Exception: pass # would print: # -# got some TypeErrors: 2 +# got some TypeErrors: [TypeError('b'), TypeError('c')] ``` Iteration over an `ExceptionGroup` that has nested `ExceptionGroup` objects @@ -338,9 +338,12 @@ print( # [ValueError('a'), TypeError('b'), TypeError('c'), KeyError('d')] ``` -### "raise e" and "raise" +### Re-raising ExceptionGroups -There is no difference between bare `raise` and a more specific `raise e`: +It is important to point out that the `ExceptionGroup` bound to `e` is an +ephemeral object. Raising it via `raise` or `raise e` will not cause changes +to the overall shape of the `ExceptionGroup`. Any modifications to it will +likely get lost: ```python try: @@ -353,25 +356,75 @@ try: ) ) except *TypeError as e: - raise # or "raise e" + e.foo = 'bar' + # ^----------- `e` is an ephemeral object that might get + # destroyed after the `except*` clause. +``` + +If the user wants to "flatten" the tree, they can explicitly create a new +`ExceptionGroup` and raise it: + + +```python +try: + raise ExceptionGroup( + ValueError('a'), + TypeError('b'), + ExceptionGroup( + TypeError('c'), + KeyError('d') + ) + ) +except *TypeError as e: + raise ExceptionGroup(*e) # would terminate with: # # ExceptionGroup( # ValueError('a'), -# TypeError('b'), # ExceptionGroup( +# TypeError('b'), # TypeError('c'), +# ), +# ExceptionGroup( # KeyError('d') # ) # ) ``` -It is important to point out that the `ExceptionGroup` bound to `e` is an -ephemeral object. Raising it via `raise` or `raise e` will not cause changes -to the overall shape of the `ExceptionGroup`. If the user wants to "flatten" -the tree, they can explicitly create a new `ExceptionGroup` and raise it: +With the regular exceptions, there's a subtle difference between bare `raise` +and a more specific `raise e`: + +```python +def foo(): | def foo(): + try: | try: + 1 / 0 | 1 / 0 + except ZeroDivisionError as e: | except ZeroDivisionError: + raise e | raise + | +foo() | foo() + | +Traceback (most recent call last): | Traceback (most recent call last): + File "/Users/guido/a.py", line 7 | File "/Users/guido/b.py", line 7 + foo() | foo() + File "/Users/guido/a.py", line 5 | File "/Users/guido/b.py", line 3 + raise e | 1/0 + File "/Users/guido/a.py", line 3 | ZeroDivisionError: division by zero + 1/0 | +ZeroDivisionError: division by zero | +``` + +This difference is preserved with exception groups: +* The `raise` form re-raises all exceptions from the group *without recording + the current frame in their tracebacks*. + +* The `raise e` form re-raises all exceptions from the group with tracebacks + updated to point out to the current frame, effectively resulting in user + seeing the `raise e` line in their tracebacks. + +That said, both forms would not affect the overall shape of the exception +group: ```python try: @@ -384,17 +437,15 @@ try: ) ) except *TypeError as e: - raise ExceptionGroup(*e) + raise # or "raise e" -# would terminate with: +# would both terminate with: # # ExceptionGroup( # ValueError('a'), +# TypeError('b'), # ExceptionGroup( -# TypeError('b'), # TypeError('c'), -# ), -# ExceptionGroup( # KeyError('d') # ) # ) From e3a383ea99fb167b342c217dc7c5ae2522f5c5b7 Mon Sep 17 00:00:00 2001 From: Yury Selivanov Date: Tue, 27 Oct 2020 09:30:25 -0700 Subject: [PATCH 8/8] Clarify the process of re-attaching a group via __context__ --- except_star.md | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-) diff --git a/except_star.md b/except_star.md index bad82d1..299992e 100644 --- a/except_star.md +++ b/except_star.md @@ -145,10 +145,11 @@ to handle, and then: * A new empty "result" `ExceptionGroup` would be created by the interpreter. -* Every `except *` clause, run from top to bottom, can filter some of the - exceptions out of the group and process them. If the except block raises an - exception, that exception is added to the "result" `ExceptionGroup` (with the - group of unprocessed exceptions referenced via the `__context__` attribute.) +* Every `except *` clause, run from top to bottom, can match a subset of the + exceptions out of the group forming a "working set" of errors for the current + clause. If the except block raises an exception, that exception is added + to the "result" `ExceptionGroup` (with its "working set" of errors + linked to that exception via the `__context__` attribute.) * After there are no more `except*` clauses to evaluate, there are the following possibilities: