Skip to content

Testing guide, DROP by name (F70) and older SQLite versions in CI - #137

Open
qiaoyuang wants to merge 7 commits into
release/2.7.0from
feature/new-features-2.7.0
Open

qiaoyuang wants to merge 7 commits into
release/2.7.0from
feature/new-features-2.7.0

Conversation

@qiaoyuang

@qiaoyuang qiaoyuang commented Nov 14, 2026 •

Copy link
Copy Markdown
Collaborator

The first part of 2.7.0, from feature/new-features-2.7.0: a guide to testing, dropping a table or a view by its name (F70), and the native tests on older SQLite versions in CI, whose SQLite libraries CI now builds. Every item has its entry in CHANGELOG.md. This is the first run of the CI changes, which can't run on a feature branch.

Roadmap and documentation

  • F70 and F71 join the roadmap (04995f2). Both come from MyMovie's move to 2.6.0: F70, High, to drop what has no class any more in a migration; F71, Low, reads that depend on a page in pagingSource, for a one-to-many relation, which no user needs yet.
  • A guide to testing, in English and Chinese, against a real SQLite on each platform (bf9c299). What it says was measured first: Robolectric's SQLite is 3.32.2 at every SDK level up to 34, 3.44.3 at 36 and 3.50.6 at 37, so the SDK level chooses the framework code, not the SQLite version; the --add-exports JVM argument is needed at SDK 36 and 37; each Robolectric test has a data directory of its own; and sqllin-processor added to kspJvmTest or kspAndroidHostTest generates the tables of the @DBRow classes of the test source sets, which only the platform's test source set can use. It ends with a recipe for testing a migration.
  • The guide to getting started says that the createSQL of a table with a converted column is computed when it's read; the guide to advanced queries, that a page of pagingSource is one SELECT, and what to do with a one-to-many relation (bf9c299).

New feature

  • F70: DROP_TABLE(name), DROP_TABLE_IF_EXISTS(name), DROP_VIEW(name) and DROP_VIEW_IF_EXISTS(name), experimental, so that a migration can drop a table or a view whose class is gone, as DROP_INDEX(name) and DROP_TRIGGER(name) do an index and a trigger. The kind is in the name, as SQLite's DROP TABLE fails for a view. (7958510)

Fixes

  • B1: PRAGMA_COMPILE_OPTIONS() failed with "This SQLite doesn't have PRAGMA compile_options" on a SQLite before 3.16.0 that has the PRAGMA, whose column is named compile_option there, rather than compile_options, by which SQLlin told whether it has it. It came with the PRAGMA in 2.6.0 (F30), and the first run of the job of SQLite 3.9.2 found it. Android's SQLite, built without the PRAGMA, still fails, as documented. (484f9ae)
  • B2: an INSERT of entities that bind more than 999 values failed with "too many SQL variables" on a SQLite before 3.32.0, as on Android before API 31, as all the values were bound to one statement. It now runs in batches of at most 999 values, whatever the SQLite, in a transaction, so that it inserts all or none, as one statement does, but INSERT_OR_FAIL, which keeps the rows before the one that fails. Its SQL, which a trigger writes into its body, is still one statement. An IN of a list that long still fails there, which its documentation now says, and the roadmap has F72, Low, to write such a list as literals. (47f6641)

CI

  • The SQLite libraries are built in CI (292d7e8). libs/linux/libsqlite3.a and libs/windows/libsqlite3.a are removed. scripts/build-sqlite.sh builds them from the sources of sqlite.org, whose SHA-256 it checks, with zig; for 3.51.1 it gives the bytes of the removed ones. The action .github/actions/sqlite caches them, and builds them on a miss with the zig of PyPI. On Windows, the tests now link the library in libs before that of MSYS2.
  • The native tests run on older SQLite versions (3582647): the new job test-on-older-sqlite runs those of sqllin-driver and sqllin-dsl on Linux, linked with SQLite 3.9.2, 3.18.2, 3.22.0 and 3.28.0, those of Android API 24, 26, 28 and 30, which no test ran on, and checks that the test binary holds that version. The tests of what depends on the version now ask for it: RENAME COLUMN, the tree of a query plan, the CHECK constraints that integrity_check verifies, a rename that rewrites the views, and the date and time functions in an index, which the guide to SQL functions now says.

Known issue, not fixed here

The test goes around it, marked KNOWN ISSUE; whether and how to fix it is to be decided.

  • Before SQLite 3.20.0, Android 7.0 to 8.1, the result columns of a join that reads a view, or a derived table that SQLite flattens, are named with their tables, as person_with_id.name, so SQLlin, which looks them up as name, can't read the rows. testSubquery runs that join from 3.20.0 on.

Testing

  • Locally, sqllin-dsl-test passes jvmTest (119 tests) and testAndroidHostTest on Robolectric API 26 and 37 (238), and sqllin-driver its JVM (13) and Android (27) tests.
  • The native tests haven't run. The development machine is an Intel Mac without any VM. They are compiled for macosArm64, linuxX64 and mingwX64, and the Linux test binaries are linked with SQLite 3.9.2.
  • What the job of older versions would meet was looked for on the JVM, with the SQLite of sqlite-jdbc 3.14.2 to 3.28.0; what is left there is those releases' own. Six tests stop at such a message on 3.14.2, so the job of 3.9.2 may still find more.
  • The first run of CI passed every job, the Windows one, which builds its library with Git Bash, Python and the zig of PyPI, included, but that of 3.9.2, which failed in testMorePragmas on B1. With B1, testMorePragmas passes on the JVM with sqlite-jdbc 3.15.1, where it failed so without it; the rest of it, after compile_options, hasn't run on 3.9.2 yet. testInsertManyRows of B2 fails with "too many SQL variables" without the fix on sqlite-jdbc 3.14.2.1, whose SQLite takes 999 parameters, and passes with it.

🤖 Generated with Claude Code

qiaoyuang and others added 7 commits November 12, 2026 23:32
Both come from MyMovie's move to 2.6.0.

F70, High: drop a table or a view by its name. Its migration drops a
table whose class is gone, which DROP_IF_EXISTS can't name, so it
borrows another table's withName. DROP_INDEX and DROP_TRIGGER already
take names; the kind goes in the name, DROP_TABLE and DROP_VIEW, as a
bare name can't tell which it is, and DROP TABLE fails on a view.

F71, Low: reads that depend on a page in the PagingSource of F13, for
a one-to-many relation that queries need as rows. MyMovie no longer
needs it, having made its relation a converted column, and such a
source can be written on the tokens of F66 today, so it waits for a
user. The item records what the discussion settled: the reads are for
a whole page and return an item for each row, their tokens join the
page's, and they must not be skipped for what the page holds.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
The guide to testing, in English and Chinese, is about testing against
a real SQLite on each platform. What it says was measured first:

- Robolectric 4.17 runs SQLite 3.32.2 at every SDK level up to 34,
  3.44.3 at 36 and 3.50.6 at 37, so the level of @config chooses the
  framework code, not the SQLite version: a statement that needs a newer
  SQLite passes at sdk 24, and DROP COLUMN fails at sdk 34. What depends
  on the version needs an emulator or a device.
- Without --add-exports=java.base/jdk.internal.access, the tests of SDK
  36 and 37 fail on JDK 21, and those of 34 and lower pass.
- Robolectric gives each test a data directory of its own, so a host
  test only has to close its database, where a JVM or native test has to
  delete the file too.
- sqllin-processor added to kspJvmTest or kspAndroidHostTest generates
  the tables of the @DBRow classes of the test source sets, and only
  those. The code that uses such a table has to be in the platform's
  test source set: in commonTest it fails with an unresolved reference,
  as the table belongs to the platform's compilation.
- observe in the test's own context, with runCurrent, shows that nothing
  is emitted, and PRAGMA_TABLE_INFO of a table that doesn't exist returns
  an empty list, which the recipe for a migration test checks with. Both
  ran as a test on the JVM.

The guide to getting started now says that the createSQL of a table
with a converted column is computed when it is read, so the generated
code holds no CREATE statement to search for. The guide to advanced
queries says that a page of pagingSource is one SELECT, what to do with
a one-to-many relation, and not to skip its read for what a page holds,
or its table is in no token.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
A migration drops a table whose class is gone, for which there is no
table object to give to DROP or DROP_IF_EXISTS. MyMovie's upgrade to
its version 3 had to borrow another table's withName to name one.

DROP_TABLE(name), DROP_TABLE_IF_EXISTS(name), DROP_VIEW(name) and
DROP_VIEW_IF_EXISTS(name) take the name, as DROP_INDEX and DROP_TRIGGER
do, and are experimental. The kind is in the function's name, as a bare
name can't tell a table from a view, and SQLite's DROP TABLE fails for a
view, and DROP VIEW for a table, with IF EXISTS too. A blank name is
rejected as the statement is built, as withName rejects one, so that no
statement of its scope runs. The statements are the ones DROP builds, so
they are schema changes to the observed queries, as before.

testDropByName covers a migration that drops a table and a view by their
names, what doesn't exist, the other kind, an FTS table and blank names.
sqllin-dsl-test passes jvmTest (118) and testAndroidHostTest (236, on
Robolectric API 26 and 37); the native tests are compiled for
macosArm64, linuxX64 and mingwX64, and not run, as this machine can't.

The guide to modifying the database, in English and Chinese, describes
them in a new section, and the guide to testing drops its table so.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
libs/linux/libsqlite3.a and libs/windows/libsqlite3.a, which the native
tests on Linux and Windows link, were binaries in the repository, built
on one machine, with nothing in the repository that told how.

scripts/build-sqlite.sh builds one from the sources of a SQLite version
that sqlite.org publishes, whose SHA-256 it checks, with zig, which
compiles for both targets on any host. It keeps the options and targets
of the libraries that were in libs: for SQLite 3.51.1 it gives their
bytes, on this machine with the zig of Homebrew and with that of PyPI,
which CI installs. It takes the autoconf sources, as sqlite.org has no
amalgamation of some patch releases, as 3.18.2, and the sqlite3.c of
both is the same file.

The composite action .github/actions/sqlite restores the library from
the cache, or builds it: a version's library only changes when the
script or the action does, which the key holds, so neither zig nor the
sources are downloaded otherwise. The Linux and Windows jobs build
3.51.1 with it. The libraries are removed and libs is ignored; whoever
runs those tests on a Linux or Windows machine runs the script.

.gitattributes keeps the script's line endings, as the checkout of a
Windows runner would make them CRLF, which bash doesn't run. On Windows
the tests now look for the library in libs before the directory of
MSYS2, as they do on Linux before the system's, so that they link the
SQLite that was built for them.

What ran here: the script, for both targets and five versions. The
workflow can only run in CI, which doesn't run on this branch.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
No test ran on a SQLite older than 3.32.2, which Robolectric has at
every SDK level up to 34, while Android has 3.9 at API 24, SQLlin's
minimum. The new job test-on-older-sqlite runs the native tests of
sqllin-driver and sqllin-dsl on Linux, linked with SQLite 3.9.2, 3.18.2,
3.22.0 and 3.28.0, those of Android API 24, 26, 28 and 30, which the
action of the commit before builds, and checks that the test binary
holds that version. It tells what those versions take, not what
Android's build of them or its framework does.

The tests of what depends on the version now ask for it, as those of
the newer PRAGMAs did: RENAME COLUMN (3.25.0), the tree of a query plan
(3.24.0), the CHECK constraints that integrity_check verifies (3.18.0),
a rename that rewrites the views that read the table (3.26.0, and a
legacy one always before 3.25.0), and the date and time functions in an
index (3.20.0), which the guide to SQL functions now says.

This machine can't run a Linux binary, so what the job will meet was
looked for on the JVM, with the SQLite of older releases of sqlite-jdbc,
3.14.2 to 3.28.0, which the tests don't run on otherwise. What is left
there is those releases' own: their error messages, and functions that
only they bundle. Six tests stop at such a message on 3.14.2, so the job
of 3.9.2 may still find more. The Linux tests link 3.9.2 here.

Two things found so that aren't fixed, and are marked KNOWN ISSUE where
the tests go around them:

- Before SQLite 3.20.0, Android 7.0 to 8.1, the result columns of a
  join that reads a view, or a derived table that SQLite flattens, are
  named with their tables, as person_with_id.name, so SQLlin, which
  looks them up as name, can't read the rows. testSubquery runs that
  join from 3.20.0 on.
- An INSERT binds the values of all its rows in one statement, and a
  SQLite before 3.32.0, Android before API 31, takes 999 parameters by
  default. testPragmas and testMorePragmas insert their 300 rows 100
  at a time.

sqllin-dsl-test passes jvmTest (118) and testAndroidHostTest (236), and
sqllin-driver its 13 and 27; the native tests are compiled.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
The first run of the job of SQLite 3.9.2 failed in testMorePragmas:
PRAGMA_COMPILE_OPTIONS() threw "This SQLite doesn't have PRAGMA
compile_options." A PRAGMA that a SQLite may lack is checked by the
name of its first column, as one that SQLite lacks returns no columns,
and before SQLite 3.16.0 the column of compile_options is named
compile_option. Measured with the SQLite of sqlite-jdbc: 3.8.11, 3.14.2
and 3.15.1 name it compile_option, and 3.16.1 and 3.18.0
compile_options; the source of 3.9.2 sets compile_option, and that of
3.18.2 the PRAGMA's name. The other PRAGMAs that SQLlin reads by name
have the same columns in 3.9.2 as in 3.51.1.

The check now takes a list of names, and compile_options both. It came
with the PRAGMA in 2.6.0 (F30). Android's SQLite is built without the
PRAGMA, which fails as before, and the Apple systems that SQLlin
supports have newer versions, so it met a SQLite that an app links on
Linux or Windows.

On the JVM with sqlite-jdbc 3.15.1, testMorePragmas fails with the
error of CI without the fix and passes with it. sqllin-dsl-test passes
jvmTest (118) and testAndroidHostTest (236); the native tests are
compiled. Whether the rest of testMorePragmas passes on 3.9.2, which
the failure kept from running, the next run of CI tells.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
An INSERT of entities bound the values of all its rows to one
statement, and a SQLite before 3.32.0, as Android's before API 31,
takes 999 parameters in a statement by default, so that 334 rows of 3
columns failed there with "too many SQL variables". The job of SQLite
3.9.2 would have met it in testPragmas, which inserts 300 rows of 4.

Where the rows bind more than 999 values, the statement now runs as
batches of rows that bind at most 999 each, whatever the SQLite: the
limit is a compile-time option that a connection can lower, which not
every platform can read, and the error has no code of its own to retry
on. The batches run in a transaction, so that all the rows are inserted
or none, as one statement does, nested in the scope's own; but those of
INSERT OR FAIL, which keeps the rows a statement inserted before the one
that fails, and so the batches before. The statement's SQL is still one
statement of all the rows, which a trigger writes into its body, where
nothing is bound, and which the log shows. A row is never split.

An IN of a list binds its values too, and can't be split, as ORDER BY,
LIMIT and aggregates would change. Its documentation, and the guide to
advanced queries, now say that a list of more than 999 values fails
before SQLite 3.32.0, and the roadmap has F72, Low, to write such a
list into the SQL as literals.

testInsertManyRows inserts 12,000 rows of 3 columns, 36,000 values, more
than any SQLite takes by default, and 400 rows whose last conflicts, in
the second batch: INSERT and INSERT_OR_ABORT insert none of them,
INSERT_OR_FAIL keeps 399, INSERT_OR_IGNORE skips the last, and
INSERT_OR_ROLLBACK in a transaction undoes the statement before too.
Without the transaction the first batch's 333 rows stay, and with one
around INSERT OR FAIL its 399 rows go, which the test fails on. On the
JVM with sqlite-jdbc 3.14.2.1, whose SQLite takes 999 parameters, it
fails with "too many SQL variables" without the fix and passes with it;
newer releases of sqlite-jdbc raise the limit. testPragmas and
testMorePragmas insert their 300 rows in one INSERT again.

sqllin-dsl-test passes jvmTest (119) and testAndroidHostTest (238); the
native tests are compiled.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant