Skip to content

Improve some sqlite3 errors #89022

Description

@serhiy-storchaka
BPO 44859
Nosy @berkerpeksag, @serhiy-storchaka, @JelleZijlstra, @erlend-aasland
PRs
  • bpo-44859: Improve error handling in sqlite3 and change some errors #27654
  • bpo-44859: Raise more accurate exceptions in sqlite3 #27695
  • Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = None
    closed_at = None
    created_at = <Date 2021-08-07.14:14:16.688>
    labels = ['extension-modules', '3.11']
    title = 'Improve some sqlite3 errors'
    updated_at = <Date 2022-03-17.05:58:28.684>
    user = 'https://github.com/serhiy-storchaka'

    bugs.python.org fields:

    activity = <Date 2022-03-17.05:58:28.684>
    actor = 'JelleZijlstra'
    assignee = 'none'
    closed = False
    closed_date = None
    closer = None
    components = ['Extension Modules']
    creation = <Date 2021-08-07.14:14:16.688>
    creator = 'serhiy.storchaka'
    dependencies = []
    files = []
    hgrepos = []
    issue_num = 44859
    keywords = ['patch']
    message_count = 8.0
    messages = ['399183', '399185', '399187', '399211', '399241', '399289', '399441', '415388']
    nosy_count = 4.0
    nosy_names = ['berker.peksag', 'serhiy.storchaka', 'JelleZijlstra', 'erlendaasland']
    pr_nums = ['27654', '27695']
    priority = 'normal'
    resolution = None
    stage = 'patch review'
    status = 'open'
    superseder = None
    type = None
    url = 'https://bugs.python.org/issue44859'
    versions = ['Python 3.11']

    Activity

    1. serhiy-storchaka commented on Aug 7, 2021

      @serhiy-storchaka
      MemberAuthor
      • MemoryError is now raised instead of sqlite3.Warning when memory is not enough for encoding statement to UTF-8 in Connection.__call__() and Cursor.execute().
      • UnicodEncodeError is now raised instead of sqlite3.Warning when statement contains surrogate characters in Connection.__call__() and Cursor.execute().
      • TypeError is now raised instead of ValueError for non-string script in Cursor.execute().
      • ValueError is now raised for script containing NUL instead of truncating it in Cursor.execute().
      • Correctly handled exceptions raised when getting boolean value of the result of the progress handler to bool.

      Also added many tests which cover different exceptional cases.

    2. serhiy-storchaka commented on Aug 7, 2021

      @serhiy-storchaka
      MemberAuthor

      There is also a problem with the sqlite3.InterfaceError raised when bind parameters with message "Error binding parameter XXX - probably unsupported type." It is raised not only for unsupported types, but:

      • For too large integers (outside of the range of signed 64-bit integers).
      • For too large strings and blobs with size <= INT_MAX but larger than the SQLite limit (1_000_000_000 by default). But OverflowError is raised for size >= INT_MAX.
      • When memory error or any other error occurred inside SQLite.
    3. serhiy-storchaka commented on Aug 7, 2021

      @serhiy-storchaka
      MemberAuthor

      Entries 3 and 4 are about Cursor.executescript(), not Cursor.execute().

    4. serhiy-storchaka commented on Aug 8, 2021

      @serhiy-storchaka
      MemberAuthor

      New changeset 0eec627 by Serhiy Storchaka in branch 'main':
      bpo-44859: Improve error handling in sqlite3 and and raise more accurate exceptions. (GH-27654)
      0eec627

    5. erlend-aasland commented on Aug 8, 2021

      @erlend-aasland
      Contributor
      • sqlite3.ProgrammingError is raised for SQLITE_MISUSE. IMO, sqlite3.InterfaceError, is more appropriate.
      • In case of too large integer values, OverflowError is raised on bind, but sqlite3.DataError is raised on UDF return. Using sqlite3.DataError in both cases would be more aligned with PEP-249.
      • Too long strings/blobs raise OverflowError. Consider raising sqlite3.DataError instead.
      • For non-contiguous (blob) buffers, we overwrite the BufferError with ValueError. Instead of overwriting BufferError, we should consider chaining with sqlite3.DataError.
      • Consider raising sqlite3.Warning when nan is implicitly converted to NULL by SQLite
      • In pysqlite_statement_create(), consider raising sqlite3.ProgrammingError iso. sqlite3.Warning when more than one statement is executed
      • In pysqlite_statement_create(), consider raising sqlite3.DataError iso. ValueError for queries with null characters
    6. erlend-aasland commented on Aug 9, 2021

      @erlend-aasland
      Contributor

      Also, for SQLite 3.7.17 and above, we should raise sqlite3.Warning for error codes SQLITE_NOTICE and SQLITE_WARNING.

    7. erlend-aasland commented on Aug 12, 2021

      @erlend-aasland
      Contributor

      Also, for SQLite 3.7.17 and above, we should raise sqlite3.Warning for error codes SQLITE_NOTICE and SQLITE_WARNING.

      Er, forget that. Neither are returned by any SQLite C interface, at the moment. They can (currently) only be used as the first argument to sqlite3_log().

    8. JelleZijlstra commented on Mar 17, 2022

      @JelleZijlstra
      Member

      New changeset 4674fd4 by Erlend Egeberg Aasland in branch 'main':
      bpo-44859: Raise more accurate exceptions in sqlite3 (GH-27695)
      4674fd4

    9. transferred this issue fromon Apr 10, 2022
    10. added a commit that references this issue on Apr 15, 2022
    11. added a commit that references this issue on May 4, 2022
    12. erlend-aasland commented on May 18, 2022

      @erlend-aasland
      Contributor
    13. added
      pendingThe issue will be closed if no feedback is provided
      on May 18, 2022
    14. erlend-aasland commented on May 19, 2022

      @erlend-aasland
      Contributor

      Thanks for bringing this up and pushing it, Serhiy.

    15. removed
      pendingThe issue will be closed if no feedback is provided
      on May 19, 2022
    16. erlend-aasland commented on Jun 1, 2022

      @erlend-aasland
      Contributor

      There is one inconsistency remaining:

      sqlite3.Cursor.execute raise sqlite3.Warning instead of sqlite3.ProgrammingError if multiple SQL statements are supplied. IMO, we should fix this.

      Actually, it is not raised in the execute method implementation; it is raised during statement creation in statement.c.

      Forget it; I already fixed this ☕ ☕ ☕

    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

    Metadata

    Metadata

    Assignees

    No one assigned

      Projects

      No projects

        Milestone

        No milestone

        Relationships

        None yet

        Development

        No branches or pull requests

        Issue actions