Repository navigation
argparse does not accept options taking arguments beginning with dash (regression from optparse) #53580
Description
Activity
andersk commented
on Jul 22, 2010 anderskmannequinMannequinAuthorPorting the a2x program to argparse from the now-deprecated optparse subtly breaks it when certain options are passed:
$ a2x --asciidoc-opts --safe gitcli.txt $ ./a2x.argparse --asciidoc-opts --safe gitcli.txt usage: a2x [-h] [--version] [-a ATTRIBUTE] [--asciidoc-opts ASCIIDOC_OPTS] [--copy] [--conf-file CONF_FILE] [-D PATH] [-d DOCTYPE] [--epubcheck] [-f FORMAT] [--icons] [--icons-dir PATH] [-k] [--lynx] [-L] [-n] [-r PATH] [-s] [--stylesheet STYLESHEET] [--safe] [--dblatex-opts DBLATEX_OPTS] [--fop] [--fop-opts FOP_OPTS] [--xsltproc-opts XSLTPROC_OPTS] [-v] a2x: error: argument --asciidoc-opts: expected one argumentApparently argparse uses a heuristic to try to guess whether an argument looks like an argument or an option, going so far as to check whether it looks like a negative number (!). It should _never_ guess: the option was specified to take an argument, so the following argument should always be parsed as an argument.
Small test case:
>>> import optparse >>> parser = optparse.OptionParser(prog='a2x') >>> parser.add_option('--asciidoc-opts', ... action='store', dest='asciidoc_opts', default='', ... metavar='ASCIIDOC_OPTS', help='asciidoc options') >>> parser.parse_args(['--asciidoc-opts', '--safe']) (<Values at 0x7f585142ef80: {'asciidoc_opts': '--safe'}>, []) >>> import argparse >>> parser = argparse.ArgumentParser(prog='a2x') >>> parser.add_argument('--asciidoc-opts', ... action='store', dest='asciidoc_opts', default='', ... metavar='ASCIIDOC_OPTS', help='asciidoc options') >>> parser.parse_args(['--asciidoc-opts', '--safe']) usage: a2x [-h] [--asciidoc-opts ASCIIDOC_OPTS] a2x: error: argument --asciidoc-opts: expected one argument
It seems like reasonable request to me to be able to allow such arguments, especially since optparse did and we want people to be able to use argparse as a replacement. Though in general I find argparse's default behavior more useful. Since argparse has been released, I'm thinking this still has to be a feature request, since argparse is *not* a drop-in replacement for optparse.
nelhage commented
on Jul 22, 2010 nelhagemannequinMannequinFor what it's worth, I have trouble seeing this as anything but a bug. I understand the motivation of trying to catch user errors, but in doing so, you're breaking with the behavior of every other option parsing library that I'm aware of, in favor of an arbitrary heuristic that sometimes guesses wrong. That's not the kind of behavior I expect from my Python libraries; I want them to do what I ask them to, not try to guess what I probably meant.
andersk commented
on Jul 22, 2010 anderskmannequinMannequinAuthorThough in general I find argparse's default behavior more useful.
I’m not sure I understand. Why is it useful for an option parsing library to heuristically decide, by default, that I didn’t actually want to pass in the valid option that I passed in? Shouldn’t that be up to the caller (or up to the program, if it explicitly decides to reject such arguments)?
Keep in mind that the caller might be another script instead of a user.
Well, even if you call it a bug, it would be an argparse design bug, and design bug fixes are feature requests from a procedural point of view.
bethard commented
on Jul 23, 2010 bethardmannequinMannequinNote that the negative number heuristic you're complaining about doesn't actually affect your code below. The negative number heuristic is only used when you have some options that look like negative numbers. See the docs for more information:
http://docs.python.org/library/argparse.html#arguments-containing
Your problem is that you want "--safe" to be treated as a positional argument even though you've declared it as an option. Basically there are two reasonable interpretations of this situation. Consider something like "--conf-file --safe". Either the user wants a conf file named "--safe", or the user accidentally forgot to type the name of the conf file. Argparse assumes the latter, though either one is conceivable. Argparse assumes the latter because, while it occasionally throws an unnecessary exception, the other behavior would allow an error to pass silently.
I'm definitely opposed to changing the default behavior to swallow some errors silently. If you'd like to propose an API for enabling such behavior explicitly and supply a patch and tests implementing it, I'll be happy to review it though.
andersk commented
on Jul 23, 2010 anderskmannequinMannequinAuthorNote that the negative number heuristic you're complaining about
doesn't actually affect your code below.Yes it does:
>>> import argparse >>> parser = argparse.ArgumentParser(prog='a2x') >>> parser.add_argument('--asciidoc-opts', ... action='store', dest='asciidoc_opts', default='', ... metavar='ASCIIDOC_OPTS', help='asciidoc options') >>> parser.parse_args(['--asciidoc-opts', '-1']) Namespace(asciidoc_opts='-1') >>> parser.parse_args(['--asciidoc-opts', '-one']) usage: a2x [-h] [--asciidoc-opts ASCIIDOC_OPTS] a2x: error: argument --asciidoc-opts: expected one argument
Your problem is that you want "--safe" to be treated as a positional
argument even though you've declared it as an option.No, it doesn’t matter whether --safe was declared as an option: argparse rejected it on the basis of beginning with a dash (as I demonstrated in my small test case, which did not declare --safe as an option, and again in the example above with -one).
Either the user wants a conf file named "--safe", or the user
accidentally forgot to type the name of the conf file.But it’s not argparse’s job to decide that the valid option I passed was actually a typo for something invalid. This would be like Python rejecting the valid call
shell = "bash"
p = subprocess.Popen(shell)
just because shell happens to also be a valid keyword argument for the Popen constructor and I might have forgotten to specify its value.Including these special heuristics by default, that (1) are different from the standard behavior of all other option parsing libraries and (2) interfere with the ability to pass certain valid options, only leads to strange inconsistencies between command line programs written in different languages, and ultimately makes the command line harder to use for everyone. The default behavior should be the standard one.
bethard commented
on Jul 26, 2010 bethardmannequinMannequinI still disagree. You're giving the parser ambiguous input. If a parser sees "--foo --bar", and "--foo" is a valid option, but "--bar" is not, this is a legitimately ambiguous situation. Either the user really wanted "--bar", and the parser doesn't support it, or the "--bar" was meant to be the argument to the "--foo" flag. At this point, the parser must make an arbitrary decision, and argparse chooses the interpretation that the user wanted the "--bar" flag.
I understand that you have a good use case for the other interpretation. That's why I suggest you come up with a patch that allows this other interpretation to be enabled when necessary. Changing the default behavior is really a non-starter unless you can propose a sensible transition strategy (as is always necessary for changing APIs in backwards incompatible ways).
andersk commented
on Jul 26, 2010 anderskmannequinMannequinAuthorI still disagree. You're giving the parser ambiguous input. If a
parser sees "--foo --bar", and "--foo" is a valid option, but "--bar"
is not, this is a legitimately ambiguous situation.There is no ambiguity. According to the way that every standard option parsing library has worked for decades, the parser knows that --foo takes an argument, so the string after --foo is in a different grammatical context than options are, and is automatically interpreted as an argument to --foo. (It doesn’t matter whether that string begins with a dash, is a valid argument, might become a valid argument in some future version, looks like a negative number, or any other such condition.)
arguments = *(positional-argument / option) [-- *(positional-argument)]
positional-argument = string
option = foo-option / bar-option
foo-option = "--foo" string
bar-option = "--bar"This is just like how variable names in Python are in a different grammatical position than keyword argument names, so that Popen(shell) is not confused with Popen(shell=True). This is not ambiguity; it simply follows from the standard definition of the grammar.
argparse’s alternative interpretation of that string as another option does not make sense because it violates the requirement that --foo has been defined to take an argument.
The only justification for considering that input ambiguous is if you start assuming that argparse knows better than the user (“the user accidentally forgot to type the name of the conf file”) and try to guess what they meant. This violates the user’s expectations of how the command line should work. It also creates subtle bugs in scripts that call argparse-based programs (think about call(["program", "--foo", foo_argument]) where foo_argument comes from some complex computation or even untrusted network input).
Changing the default behavior is really a non-starter unless you can
propose a sensible transition strategy (as is always necessary for
changing APIs in backwards incompatible ways).This would not be a backwards incompatible change, since every option that previously parsed successfully would also parse in the same way after the fix.
andersk commented
on Jul 26, 2010 anderskmannequinMannequinAuthorarguments = *(positional-argument / option) [-- *(positional-argument)]
positional-argument = string
option = foo-option / bar-option
foo-option = "--foo" string
bar-option = "--bar"Er, obviously positional arguments before the first ‘--’ can’t begin with a dash (I don’t think there’s any confusion over how those should work).
arguments = *(non-dash-positional-argument / option) ["--" *(positional-argument)]
non-dash-positional-argument = <string not beginning with "-">
positional-argument = stringThe point was just that the grammar unambiguously allows the argument of --foo to be any string.
bethard commented
on Jul 27, 2010 bethardmannequinMannequinIt *would* be a backwards incompatible change. Currently, if I have a parser with both a "--foo" and a "--bar" option, and my user types "--foo --bar", they get an error saying that they were missing the argument to "--foo". Under your proposal, the "--foo" option will now silently consume the "--bar" option without an error. I know this is good from your perspective, but it would definitely break some of my scripts, and I imagine it would break other people's scripts as well.
As I keep saying, I'm happy to add your alternative parsing as an option (assuming you provide a patch), but I really don't think it's the right thing to do by default. Most command line programs don't have options that take other option-like things as arguments (which is the source of your problem), so in most command line programs, people want an error when they get an option they don't recognize or an option that's missing its argument. Under your proposal, more such errors will pass silently and will have to be caught by additional code in the script.
drm commented
on Feb 5, 2011 drmmannequinMannequinThe reporter imho is 100% right. Simply because of the fact that in the current situation, there is no way to supply an argument starting with a dash (not even for instance a filename). That is, of course, total nonsense to be dictated by the parser library.
Reacted by christophe grosjean, Alex Díaz and Tom SchraitleWhile I also dislike the existing behavior, note that you can get what you want by using an equal sign.
>>> import argparse >>> parser = argparse.ArgumentParser(prog='a2x') >>> parser.add_argument('--asciidoc-opts', ... action='store', dest='asciidoc_opts', default='' ... metavar='ASCIIDOC_OPTS', help='asciidoc options') >>> parser.parse_args(['--asciidoc-opts', '-1']) Namespace(asciidoc_opts='-1') >>> parser.parse_args(['--asciidoc-opts=-one']) Namespace(asciidoc_opts='-one')
I always use the equal sign, so I've never noticed this behavior before.
I wish that help would display the equal sign, but that's another issue.
78 remaining items
If there's really no way to make this work as desired in general, could there instead be either an action, or an nargs value, that means "for this option, take the next argv element as the option's argument no matter what it looks like"? Like how
grep's-eoption works.I don't think that's possible due to the way argparse scans the entire argv list before it starts processing.
@ncoghlan would you say this is a bug in argparse, or a consequence of fundamental differences between optparse and argparse? If optparse is still available and there is a workaround for argparse (using = rather than space), then a doc-only fix could close this?
"Fixing" the problem with documentation isn't really a fix. The documentation you need to fix isn't the documented targeted as people writing
argparseprograms, but people invoking them (perhaps without knowing they're even written in Python). You have to teach the whole world to get in the habit of writing--foo=ARGinstead of--foo ARGjust in case they're invoking a program that happens to useargparse. That doesn't seem right.I've never understood what's supposed to be the fundamental algorithmic impossibility of fixing this bug. I'm sure TIWTOWTDI, but one simple approach that I've used in personal projects is to preprocess argv and convert
--foo barto--foo=barin appropriate context tracked with a little state machine. Granted, I didn't try to handle oddities likenargsnot in0, 1, REMAINDERbut it works fine overall and solves the problem for the case of mandatory arguments to regular options.one simple approach that I've used in personal projects is to preprocess argv and convert
--foo barto--foo=barin appropriate context tracked with a little state machine. Granted, I didn't try to handle oddities likenargsnot in0, 1, REMAINDERbut it works fine overall and solves the problem for the case of mandatory arguments to regular options.It doesn't work if you want to do
--foo=--. argparse has weird, very non-intuitive edge cases like that.In the meantime optparse has been un-deprecated because of the argparse problems (see #126180 ), so I'm fine with just switching to optparse.
Reacted by Alyssa CoghlanIt doesn't work if you want to do --foo=--. argparse has weird, very non-intuitive edge cases like that.
Also fixable. All I'm saying is that I'm not quite convinced that this class of problem is algorithmically unfixable in argparse.
Yes, this is a consequence of fundamental differences between argparse and all other parsers. The only way to solve this and many other issues is to implement an alternative parsing algorithm, and make it default, but keep an option to use the old algorithm (or several options to enable particular parts of the old algorithm). I am working on this since last year. This is not easy, because there are many features and many flexibility in argparse (they are not always work together in the current argparse). We need to keep as much compatibility as possible in non-corner cases.
Reacted by Éric and Alyssa CoghlanFrom a documentation point of view, the updated
optparseintroduction (added when the deprecation was reverted) already cites "the application requires additional control over the handling of options which accept parameter values that may start with - (such as delegated options to be passed to invoked subprocesses)" as one potential reason for choosing optparse over argparse.I think the argparse feature request is still valid, though. Falling back to optparse is just an option that's available now, rather than one that may become available in some future version of argparse.
I use the binding operator and double quotes to force the issue, for example:
--ssh-options="--foo --bar"This gives me the string "--foo --bar" which you can then pass to ssh.
This works because argparse does not separate the binding operator parts until later. The reason for argparse to act this way is to support intermixed arguments which are tricky to handle.
One option, if you truly love optparse so much, is to copy the module from the Python packages and put it into your package as is, with a name change such as optparse312.py.
Metadata
Metadata
Assignees
Projects
- Status
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
bugs.python.org fields: