Skip to content

PEP 689 -- Add an unstable C-API tier #91744

Closed
Closed
PEP 689 -- Add an unstable C-API tier#91744
Assignees
encukou
Labels
topic-C-API

Description

@encukou
encukou
opened on Apr 20, 2022

On python-dev, it became clear that it would be useful to have a “semi-stable” “unstable” C-API tier which will stay stable within a minor release.

  • This API will go in a new directory (Include/unstable/) - see Nick's reply
    You'll need to #define Py_USING_UNSTABLE_API.
    (name still up for bikeshedding).

  • Since we're nearing Beta and there's no rush to break things, in 3.11
    you only get a warning if you try to use it without the opt-in #define.
    In 3.12 it'll fail.

  • The functions will be renamed to drop the leading underscore. The old
    names will be available, and may be removed
    whenever the API changes. (Ideally, the underscore should always mark
    API that's fully private with no guarantees at all.)

  • Get SC approval (this is now PEP-689 that will need acceptance)

  • The API will be stable during a minor release. (As usual, for extreme
    cases, exceptions are possible with SC approval.)

  • Docs:

  • All ununstable API should be tested

    • PEP-523 PyCode_* functions: tested in test_code
    • ...

What should be here:

  • Functions added in PEP-523
  • PyCode_New, PyCode_NewWithPosOnlyArgs
  • Ideally anything documented as subject to change between minor
    releases. (To be kind to users, if something is added later we should
    again have one release of compiler warnings before requiring the opt-in.
    Unless that API just changed and users would get errors anyway.)
  • Cinder's dict writing API looks like it would feel at home here: add support for watching writes to selected dictionaries #91052 (comment)
  • PyBytesObject.ob_shash sounds like a good candidate: see https://discuss.python.org/t/15108
  • non-opaque access to frame structs and any other key APIs needed to
    implement alternate eval loops with comparable performance to the default
    eval loop (unless & until we can figure out stable public APIs that can
    deliver equivalent performance) - see Nick's reply
  • C APIs that provide access to compiled code whether in AST or opcode
    form (the API itself may be stable, but the compiled code isn't, so this is
    kinda covered by your last point) - see Nick's reply
  • _Py_HashDouble, see comment below

First PR: #91789

Activity

  1. encukou
    added
    topic-C-API
    on Apr 20, 2022
  2. encukou
    self-assigned this
    on Apr 20, 2022

  3. vstinner commented on Apr 20, 2022

    @vstinner
    vstinner
    on Apr 20, 2022
    Member

    This API will go in a new directory (Include/semi-stable/) - see Nick's reply

    For the directory name, I suggest to omit - or _ and just put the two words as semistable, it's easier to type.

    C compilers don't handle well header files with the same names in different directories. For Include/cpython/, all header files fail with #error if they are included directly. IMO we can do the same here to ensure that Include/semistable/ is not added to the search paths by mistake.

    Another option is to add a prefix to each file, like pycore_ for the internal C API. For example, Include/internal/pycore_object.h is related to Include/object.h. But that's more annoying and confusing.

    I would prefer that C extensions just include the main <Python.h> and that's it (no change). I would prefer that C extensions don't include directly <semistable/frame.h> for example.

  4. vstinner commented on Apr 20, 2022

    @vstinner
    vstinner
    on Apr 20, 2022
    Member

    UPDATE: I completed my list, I forgot some removed functions.

    Document the new opt-in macro

    When building Python itself, does the Py_BUILD_CORE macro imply defining the Py_USING_SEMI_STABLE_API macro?

    For me, the internal API (Py_BUILD_CORE) is a superset of semi-stable API (Py_USING_SEMI_STABLE_API).

    APIs that provide access to compiled code whether in AST or opcode form (the API itself may be stable, but the compiled code isn't, so this is kinda covered by your last point)

    In Python 3.10, Python-ast.h was moved to the internal C API as pycore_ast.h, and many API related to AST have been moved to the internal C API:

    • PyST_GetScope()
    • PySymtable_Build()
    • PySymtable_BuildObject()
    • PySymtable_Free()
    • Py_SymtableString()
    • Py_SymtableStringObject()
    • PyAST_Compile()
    • PyAST_CompileEx()
    • PyAST_CompileObject()
    • PyFuture_FromAST()
    • PyFuture_FromASTObject()
    • PyParser_ASTFromFile()
    • PyParser_ASTFromFileObject()
    • PyParser_ASTFromFilename()
    • PyParser_ASTFromString()
    • PyParser_ASTFromStringObject()
    • PyArena_New()
    • PyArena_Free()
    • PyArena_Malloc()
    • PyArena_AddPyObject()
    • see: https://docs.python.org/dev/whatsnew/3.10.html#id4

    Are there remaining "AST related" functions in the C API?

    Obviously, compile() can still be called with PyCF_ONLY_AST flag, or call ast.parse(). But this API returns Python objects, not C structures.

  5. vstinner commented on Apr 20, 2022

    @vstinner
    vstinner
    on Apr 20, 2022
    Member

    I would be nice to move the PyTypeObject to the semi-stable API, but IMO it's not possible to remove it from the default API because way too many C extensions use it. Modify all these extensions to define Py_USING_SEMI_STABLE_API is just annoying.

  6. encukou commented on Apr 20, 2022

    @encukou
    encukou
    on Apr 20, 2022
    MemberAuthor

    OK, I'll do semistable/, and I already have #error in my WIP.

    When building Python itself, does the Py_BUILD_CORE macro imply defining the Py_USING_SEMI_STABLE_API macro?

    Yes.

    Are there remaining "AST related" functions in the C API?

    @ncoghlan?

    I would be nice to move the PyTypeObject to the semi-stable API

    Definitely not in 3.11.

  7. encukou commented on Apr 20, 2022

    @encukou
    encukou
    on Apr 20, 2022
    MemberAuthor

    The old underscore-prefixed names can be deprecated static inline functions, rather than #defines as in the original plan.

  8. ncoghlan commented on Apr 20, 2022

    @ncoghlan
    ncoghlan
    on Apr 20, 2022
    Contributor

    I forgot (or simply didn't notice) that the AST APIs had been migrated to the internal API.

    I still think they'd be good candidates for being declared semi-stable - part of the rationale for the new tier is reducing the temptation for 3rd parties to include the internal API headers, and these were considered useful enough to be public APIs prior to 3.10

  9. vstinner commented on Apr 20, 2022

    @vstinner
    vstinner
    on Apr 20, 2022
    Member

    Which projects use the C API of AST functions removed in Python 3.10? What is the advantage of adding back this removed API?

    These functions were undocumented, excluded from the limited C API and had no test. The removal is related to the PEG parser PEP 617 which removed the old parser. The implementation modified the C code and in C, it was no longer possible to create struct _mod (mod_ty) which were arguments of the removed AST functions.

    In Python 3.8, you needed the parser to get a node* object to then get a mod_ty object. But the whole C API of the parser has been removed in Python 3.10: the struct _node (node) type has been removed.

    In Python 3.10 and 3.11, it's possible to browse AST using the Python API. Just use the Python ast module. IMO it's enough. There is no need to add a public C API for that.

  10. vstinner commented on Apr 20, 2022

    @vstinner
    vstinner
    on Apr 20, 2022
    Member

    The old underscore-prefixed names can be deprecated static inline functions, rather than #defines as in the original plan.

    If there is no plan to deprecate the old names, macros are fine. Like the current code in Include/cpython/abstract.h:

    // Backwards compatibility aliases for API that was provisional in Python 3.8
#define _PyObject_Vectorcall PyObject_Vectorcall
#define _PyObject_VectorcallMethod PyObject_VectorcallMethod
#define _PyObject_FastCallDict PyObject_VectorcallDict
#define _PyVectorcall_Function PyVectorcall_Function
#define _PyObject_CallOneArg PyObject_CallOneArg
#define _PyObject_CallMethodNoArgs PyObject_CallMethodNoArgs
#define _PyObject_CallMethodOneArg PyObject_CallMethodOneArg

    By the way, IMO it's time to deprecate these aliases :-)

    Then nice advantage of a static inline function is that it can be decorated with Py_DEPRECATED(3.11).

  11. encukou commented on Apr 20, 2022

    @encukou
    encukou
    on Apr 20, 2022
    MemberAuthor

    Haha, _PyInterpreterState_GetEvalFrameFunc and _PyInterpreterState_SetEvalFrameFunc, are not mentioned in PEP 523 at all, but their docs say See PEP 523...

    I guess they fall under Functions added in PEP-523. Off with their underscores!

  12. vstinner commented on Apr 20, 2022

    @vstinner
    vstinner
    on Apr 20, 2022
    Member

    Haha, _PyInterpreterState_GetEvalFrameFunc and _PyInterpreterState_SetEvalFrameFunc, are not mentioned in PEP 523 at all, but their docs say See PEP 523...

    These functions should be made public in the semi-stable API. The PyInterperterState structure was made opaque in Python 3.8 (released in 2019), PEP 523 was approved in 2016. I added these functions to keep PEP 523 usable. Otherwise, you had to include the internal C API to use the PEP, hum...

  13. encukou commented on Apr 21, 2022

    @encukou
    encukou
    on Apr 21, 2022
    MemberAuthor

    @vstinner I see the function is documented to take PyFrameObject: https://docs.python.org/3.10/c-api/init.html?highlight=pyframeevalfunction#c._PyFrameEvalFunction
    But it actually takes a _PyInterpreterFrame. Should that be moved to semi-stable API too?

  14. encukou commented on Apr 21, 2022

    @encukou
    encukou
    on Apr 21, 2022
    MemberAuthor

    I found the commit that changed it and asked there: #88756 (comment)

  15. vstinner commented on Apr 21, 2022

    @vstinner
    vstinner
    on Apr 21, 2022
    Member

    If you ask me, I suggest to move these functions and type to the semi-stable API:

    • _PyFrameEvalFunction type
    • _PyInterpreterState_GetEvalFrameFunc()
    • _PyInterpreterState_SetEvalFrameFunc()
    • PyCode_New()
    • PyCode_NewWithPosOnlyArgs()

    I would prefer to not expose _PyEval_EvalFrameDefault() directly. It should be the default value of _PyInterpreterState_GetEvalFrameFunc(). But if someone considers that it should be exposed, I'm also fine with that.

  16. 5 remaining items

  17. encukou commented on Apr 22, 2022

    @encukou
    encukou
    on Apr 22, 2022
    MemberAuthor

    So, What's New was updated to say:

    :c:func:_PyFrameEvalFunction now takes _PyInterpreterFrame* as its second parameter, instead of PyFrameObject*. See :pep:523 for more details of how to use this function pointer type.

    But that's pretty confusing, PEP 523 does not mention _PyInterpreterFrame!

    If I understand correctly, _PyInterpreterFrame needs to stay private, and _PyFrameEvalFunction needs to take it so _PyEval_EvalFrameDefault can stay fast.
    But if users can access it, it needs to be exposed (as an opaque structure, of course).

    Would it be appropriate to also put _PyFrame_GetFrameObject in the semi-stable API, and say use PyFrame_GetFrameObject to convert _PyInterpreterFrame to PyFrameObject?

  18. vstinner commented on Apr 22, 2022

    @vstinner
    vstinner
    on Apr 22, 2022
    Member

    You should ask @markshannon and users of these API.

  19. encukou
    changed the title [-]Add a semi-stable C-API tier[/-] [+]PEP 689 -- Add a semi-stable C-API tier[/+] on Apr 27, 2022

  20. seberg commented on Apr 27, 2022

    @seberg
    seberg
    on Apr 27, 2022
    Contributor

    Since NumPy uses it, does it make sense to move _Py_HashDouble into the semi stable API? We also seem to use _PyDict_GetItemStringWithError, although that is easy to just ship on our own (we do for older versions anyway). This is a quick search, I may be missing things. Or maybe just put them into the public API directly?

  21. encukou commented on Apr 28, 2022

    @encukou
    encukou
    on Apr 28, 2022
    MemberAuthor

    _PyDict_GetItemStringWithError is used in only a few places in CPython, frequently for static strings. IMO it would be better for CPython to reevaluate using it.
    _Py_HashDouble looks OK for semi-stable API, but I wouldn't make it fully public -- its handling of NaN, which is the reason it takes an extra PyObject*, is a detail we might want to change.

  22. encukou
    changed the title [-]PEP 689 -- Add a semi-stable C-API tier[/-] [+]PEP 689 -- Add an unstable C-API tier[/+] on May 3, 2022

  23. jpe commented on Jul 12, 2022

    @jpe
    jpe
    on Jul 12, 2022

    I'm porting a debugger that uses a _PyFrameEvalFunction and ran into the problem of _PyInterpreterFrame being opaque (unless I use the internal headers). I think I can work around this, but for some purposes (including, I think, the motivating case for adding eval_frame of a jit compiler) access to the frame to be evaluated is needed. A variant of _PyFrame_GetFrameObject could be used or it may be possible to maintain the old signature and have a frame passed -- for performance, the core would need to bypass the eval_frame if it is set to the default

  24. gvanrossum commented on Jul 12, 2022

    @gvanrossum
    gvanrossum
    on Jul 12, 2022
    Member

    @jpe, What information do you need to get out of the frame? It may make more sense to offer APIs (stable or not) that give you the specific things you are looking for.

  25. jpe commented on Jul 12, 2022

    @jpe
    jpe
    on Jul 12, 2022

    As of now, I need the global and locals. But for something like a jit compiler (I think eval_frame was added for pyjion), you'd need more. As I mentioned I can work around this, but I think that eval_frame currently isn't very useful and either should be deprecated / removed or made to work reasonably.

  26. gvanrossum commented on Jul 12, 2022

    @gvanrossum
    gvanrossum
    on Jul 12, 2022
    Member

    If your issue is that PEP 623 isn't the right API, I think you're preaching to the choir, but I'm not sure how much PEP 689 can help -- it isn't the place to design a new API.

  27. jpe commented on Jul 12, 2022

    @jpe
    jpe
    on Jul 12, 2022

    My issue is I don't think the eval_frame code in 3.11 is very useful as it stands and think it should be either deprecated / removed or made as useful as it was in 3.10. I suspect if left as it is, it will either not be used or that internal api's will be need to be used.

  28. gvanrossum commented on Jul 12, 2022

    @gvanrossum
    gvanrossum
    on Jul 12, 2022
    Member

    Noted. Unfortunately our expert on that topic (@markshannon) is currently on vacation. I think it's too late for code changes in 3.11, so you are likely going to be using internal APIs (which might become "unstable but supported APIs" in 3.12, if PEP 689 is accepted).

  29. encukou commented on Aug 10, 2022

    @encukou
    encukou
    on Aug 10, 2022
    MemberAuthor

    I'm deferring the PEP for now.

  30. encukou
    closed this as completedon Aug 10, 2022

  31. vstinner commented on Aug 11, 2022

    @vstinner
    vstinner
    on Aug 11, 2022
    Member

    Oh, too bad, I liked this PEP. But yeah, it's a lot of work :-(

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

topic-C-API

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions