PEP 515-style formatting with underscores does not seem to work for Decimal

@tim-one

BPO	45708
Nosy	@tim-one, @rhettinger, @mdickinson, @ericvsmith, @tiran, @serhiy-storchaka, @sobolevn, @sndrtj
PRs	bpo-45708: Support underscore separators when formatting Decimal objects #29438

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2021-11-04.09:01:14.767>
labels = ['type-feature', 'library', '3.11']
title = 'PEP 515-style formatting with underscores does not seem to work for Decimal'
updated_at = <Date 2021-11-07.16:29:51.064>
user = 'https://github.com/sndrtj'

bugs.python.org fields:

activity = <Date 2021-11-07.16:29:51.064>
actor = 'eric.smith'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Library (Lib)']
creation = <Date 2021-11-04.09:01:14.767>
creator = 'sndrtj'
dependencies = []
files = []
hgrepos = []
issue_num = 45708
keywords = ['patch']
message_count = 14.0
messages = ['405667', '405679', '405683', '405684', '405685', '405724', '405739', '405791', '405795', '405853', '405854', '405855', '405895', '405907']
nosy_count = 8.0
nosy_names = ['tim.peters', 'rhettinger', 'mark.dickinson', 'eric.smith', 'christian.heimes', 'serhiy.storchaka', 'sobolevn', 'sndrtj']
pr_nums = ['29438']
priority = 'normal'
resolution = None
stage = 'patch review'
status = 'open'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue45708'
versions = ['Python 3.11']

Hello,

It appears that Decimal does not support PEP-515 style formatting with underscores as thousands separators.

>>> from decimal import Decimal
>>> f"{Decimal('5000'):,}"
'5,000'
>>> f"{Decimal('5000'):_}"
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ValueError: invalid format string

This does work for all the other types mentioned in PEP-515

>>> f"{5000:_}"
'5_000'
>>> f"{5000.0:_}"
'5_000.0'
>>> f"{complex(5000, 1):_}"
'(5_000+1j)'

I have tried this on python 3.8, 3.9 and 3.10 on a Mac.

It is not hard to fix (

cpython/Modules/_decimal/libmpdec/io.c

Lines 857 to 863 in 2c045bd

    
           /* thousands separator */ 
        
           if (*cp == ',') { 
        
               spec->dot = "."; 
        
               spec->sep = ","; 
        
               spec->grouping = "\003\003"; 
        
               cp++; 
        
           }

):

    /* thousands separator */
    if (*cp == ',' || *cp == '_') {
        spec->dot = ".";
        if (*cp == ',') {
            spec->sep = ",";
        } else {
            spec->sep = "_";
        }
        spec->grouping = "\003\003";
        cp++;
    }

But, there's some context to it: https://bugs.python.org/issue44267 / https://mail.python.org/pipermail/python-dev/2016-March/143556.html

I am going to research this further.

I think the two main reasons that applied to not implementing the parsing part of PEP-515 for the Decimal type (speed, compliance with the IBM specification) don't apply to the formatting side.

We do need to think about the implications of making local changes to our copy of the externally-maintained libmpdec library, though.

Changing Python versions: this is a new feature, so could only go into Python 3.11.

We do need to think about the implications of making local changes to our copy of the externally-maintained libmpdec library, though.

It looks like quite similar changes have already been made: 298131a

I will send a PR, so we can see what exactly it touches / changes.

It looks like quite similar changes have already been made:

Yes, I think this isn't something that needs to be resolved for this issue, but it is something we need to think about. (Though perhaps the resolution is just "Don't worry about it until we need to.")

I will send a PR, so we can see what exactly it touches / changes.

Ah, sorry; I already made one before reading your message. I'd be happy to get your input on that PR, though. (Or to review a PR from you.)

Christian Heimes pointed out in the PR discussion that we can't simply modify libmpdec, since some vendors unbundle the mpdecimal library.

So some options are:

Do nothing.
Request that this feature to be added upstream, so that it eventually makes its way into core Python.
Bypass mpd_parse_fmt_str and do our own format string parsing in _decimal.c (e.g., by copying and adapting the code in mpdecimal).
Wrap mpd_parse_fmt_str and do our own pre- and post- processing in _decimal.c (pre-process to convert "_" to "," in the format string, then post-process the formatted string to convert "," back to "_").

Option 2 makes sense to me from the point of view of separation of concerns: libmpdec aims to implement Cowlishaw's specification, and formatting lies outside of that specification. The decimal specification is pretty much set in stone, but the formatting mini-language could change again in the future, and when that happens we should be able to update the CPython code accordingly. (This brings to mind Robert Martin's Single Responsibility Principle: "Gather together those things that change for the same reason, and separate those things that change for different reasons.")

I've updated the PR (and turned it into a draft) to show what option 2 looks like. The duplication is a little ugly.

I think your option 2 makes the most sense.

Python 3.15.0a1+ (heads/main:9a87ce8b57f, Oct 17 2025, 03:17:14) [GCC 12.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from decimal import Decimal
>>> f"{Decimal('5000'):_}"
'5_000'
>>>

Now we have a fallback to pure-Python code, when mpd_parse_fmt_str() fails:

cpython/Modules/_decimal/_decimal.c

Lines 3780 to 3786 in bedaea0

    
           if (!mpd_parse_fmt_str(&spec, fmt, CtxCaps(context))) { 
        
               if (replace_fillchar) { 
        
                   PyMem_Free(fmt); 
        
               } 
        
               return pydec_format(dec, context, fmtarg, state); 
        
           }

Perhaps, we could eventually transform pydec_format() helper to C code. But that looks like another issue.

Navigation Menu

Repository navigation

PEP 515-style formatting with underscores does not seem to work for Decimal #89871

18 remaining items

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

	/* thousands separator */
	if (*cp == ',') {
	spec->dot = ".";
	spec->sep = ",";
	spec->grouping = "\003\003";
	cp++;
	}

	if (!mpd_parse_fmt_str(&spec, fmt, CtxCaps(context))) {
	if (replace_fillchar) {
	PyMem_Free(fmt);
	}

	return pydec_format(dec, context, fmtarg, state);
	}

PEP 515-style formatting with underscores does not seem to work for Decimal #89871

Description

Activity

sndrtj commented on Nov 4, 2021

sobolevn commented on Nov 4, 2021

mdickinson commented on Nov 4, 2021

18 remaining items

sobolevn commented on Nov 6, 2021

mdickinson commented on Nov 6, 2021

mdickinson commented on Nov 7, 2021

ericvsmith commented on Nov 7, 2021

skirpichev commented on Oct 19, 2025

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions