Python 3.12 breaks backwards compatibility for logging configuration

Bug report

Bug description:

This worked fine on previous versions:

import logging
import logging.config
import logging.handlers
import multiprocessing as mp


def main():
    config = {
        'version': 1,
        'handlers': {
            'sink': {
                'class': 'logging.handlers.QueueHandler',
                'queue': mp.get_context('spawn').Queue(),
            },
        },
        'root': {
            'handlers': ['sink'],
            'level': 'DEBUG',
        },
    }
    logging.config.dictConfig(config)


if __name__ == '__main__':
    main()

With Python 3.12, it drops an error:

Traceback (most recent call last):
  File "/home/egor/.pyenv/versions/3.12-dev/lib/python3.12/logging/config.py", line 581, in configure
    handler = self.configure_handler(handlers[name])
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/egor/.pyenv/versions/3.12-dev/lib/python3.12/logging/config.py", line 786, in configure_handler
    raise ValueError('No handlers specified for a QueueHandler')
ValueError: No handlers specified for a QueueHandler

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/egor/workspace/personal/cimple/../test.py", line 25, in <module>
    main()
  File "/home/egor/workspace/personal/cimple/../test.py", line 21, in main
    logging.config.dictConfig(config)
  File "/home/egor/.pyenv/versions/3.12-dev/lib/python3.12/logging/config.py", line 912, in dictConfig
    dictConfigClass(config).configure()
  File "/home/egor/.pyenv/versions/3.12-dev/lib/python3.12/logging/config.py", line 588, in configure
    raise ValueError('Unable to configure handler '
ValueError: Unable to configure handler 'sink'

More than that, even the example in the Logging Cookbook is broken now: https://docs.python.org/3/howto/logging-cookbook.html#a-more-elaborate-multiprocessing-example (fails with the same error).

Version info:

# python -VV
Python 3.12.0+ (heads/3.12:f108785, Nov  1 2023, 19:47:19) [GCC 13.2.1 20230801]

CPython versions tested on:

3.12

Operating systems tested on:

Linux

Linked PRs

Thanks for the report. This is due to a fix for gh-93162 , which allows proper configuration of a QueueListener and QueueHandler together. I'm sorry about the breakage - it appears not enough consideration was given to configuring a Queuehandler without configuring a QueueListener. This was of course the default case before the change went in, and the cookbook example needs to be updated. How are you configuring the QueueListener in your environment?

I'm using QueueListener/QueueHandler w/ multiprocessing to centralize logging in the parent process logger, which I think is common advise on the net. The complete setup is:

@contextmanager
def configure_logging_in_child(queue):
    config = {
        'version': 1,
        'handlers': {
            'sink': {
                'class': 'logging.handlers.QueueHandler',
                'queue': queue,
            },
        },
        'root': {
            'handlers': ['sink'],
            'level': 'DEBUG',
        },
    }
    logging.config.dictConfig(config)
    try:
        yield
    except Exception as e:
        logging.exception(e)

def child_main(log_queue):
    with configure_logging_in_child(log_queue):
        # Do some actual work.
        ...

def main():
    ...
    ctx = multiprocessing.get_context('spawn')
    queue = ctx.Queue()
    listener = logging.handlers.QueueListener(queue, logging.getLogger())
    listener.start()
    # There's actually a list of child processes.
    args = (queue,)
    child = ctx.Process(target=child_main, args=args)
    child.start()
    child.join()
    ...

To be honest, I don't really understand how the queue & listener can be configured in one call, since I configure them separately in the parent & child processes.

pythongh-111615: Fix regression in QueueHandler configuration.

To be honest, I don't really understand how the queue & listener can be configured in one call

Well, they can be if you're using threads rather than processes.

Well, they can be if you're using threads rather than processes.

I understand. I cannot use Python threads though, since I need actual parallel execution (for a number of child processes).

I use a similar pattern in a couple of other projects, where I also must use multiprocessing, since the child process needs to drop permissions, execute as a different user, etc.

Well, the change in the linked PR should resolve this, but it will have to wait for the next release of Python 3.12.

Well, the change in the linked PR should resolve this, but it will have to wait for the next release of Python 3.12.

Thanks, sounds good!

gh-111615: Fix regression in QueueHandler configuration. (GH-111638)

pythongh-111615: Fix regression in QueueHandler configuration. (pytho…

[3.12] gh-111615: Fix regression in QueueHandler configuration. (GH-1…

pythongh-111615: Fix regression in QueueHandler configuration. (pytho…

Navigation Menu

Repository navigation

Python 3.12 breaks backwards compatibility for logging configuration #111615

Bug report

Bug description:

CPython versions tested on:

Operating systems tested on:

Linked PRs

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Python 3.12 breaks backwards compatibility for logging configuration #111615

Description

Bug report

Bug description:

CPython versions tested on:

Operating systems tested on:

Linked PRs

Activity

vsajip commented on Nov 2, 2023

egor-tensin commented on Nov 2, 2023

vsajip commented on Nov 2, 2023

egor-tensin commented on Nov 2, 2023

vsajip commented on Nov 2, 2023

egor-tensin commented on Nov 2, 2023

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions