Repository navigation
[MRG] Read the number of plans from axis 0 in merge_bijections - #911
Conversation
perms is a stack of (n_plans, n_samples) permutations, but merge_bijections_c passed plans.shape[1] -- the number of samples -- as the plan count, so the C++ merge loop walked n_samples plans over a buffer holding n_plans. With n_plans < n_samples it read past the end of the array (heap-buffer-overflow under AddressSanitizer, segmentation fault through the public API); with n_plans > n_samples it silently dropped every plan past the n_samples-th. Take the count from plans.shape[0] and reject a perms that does not follow the documented layout at the Python boundary, so a wrong row length raises instead of reaching C++.
AddressSanitizer on the built extension still reports a heap-buffer-overflow when merge_bijections_c is called directly with rows that are not one entry per point: the C++ loop reads n_plans * n_samples integers, so a shorter row leaves it short of the buffer even with the plan count read from the right axis. Check plans.shape[1] against the number of points in the Cython layer and assert that refusal from the test as well, so the overrun is unreachable for any caller rather than only for the public wrapper.
|
Thanks for flagging this as worth its own report -- it is now #911. The root cause is one axis: The fix reads the count from Memory safety was verified on the built extension rather than a stand-in: Five tests added, all of which fail without the fix (four abort the interpreter, one fails its Left out on purpose, and stated in the PR rather than hidden: a One note for whoever merges: this branch and #898 both append to the end of CI is still running on |
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## master #911 +/- ##
==========================================
+ Coverage 97.00% 97.06% +0.06%
==========================================
Files 130 130
Lines 26995 27075 +80
==========================================
+ Hits 26187 26281 +94
+ Misses 808 794 -14 🚀 New features to boost your workflow:
|
baptiste-genest
left a comment
There was a problem hiding this comment.
Thanks, it looks good and all tests pass here, including with #898's C++ change.
One thing: invalid perms values still crash or hang. Could you please add a permutation check and reject zero plans too?
if perms.shape[0] == 0:
raise ValueError("perms must contain at least one plan")
if not np.all(np.sort(perms, axis=1) == np.arange(X_np.shape[0])):
raise ValueError("each row of perms must be a permutation of 0..n_samples-1")| cdef int n = X.shape[0] | ||
| cdef int d = X.shape[1] | ||
| cdef int k = plans.shape[1] | ||
| # plans is laid out as (n_plans, n_samples), so the number of plans to merge is |
There was a problem hiding this comment.
I don't feel like the comments are necessary here, this is what the correct implementation should have been from the start, it's my fault ^^
There was a problem hiding this comment.
Thanks! I've removed the explanatory comments and added validation for empty plan stacks and invalid permutation rows before the native merge call, including at the Cython boundary. Regression tests cover both the public API and direct Cython entry point.
merge_bijections accepted any stack whose rows were one entry per point, so a row that is not a permutation of 0..n_samples-1 reached BijectiveMatching and indexed the point clouds out of bounds, and an empty stack left the merge helper with nothing to read. Both crash the interpreter. Reject an empty plan stack and check that every row is a permutation of 0..n_samples-1, at the Python entry point and again in merge_bijections_c so a direct caller cannot reach C++ with a malformed stack. The entries are checked before the conversion to int32, which truncates fractional values and wraps integers that do not fit and would otherwise turn an invalid row into a valid one.
|
Thanks, the fix looks right! A few small things before merging: could you remove the permutation loop in merge_bijections_c and its tests, since the Python check should already cover it? and shorten the RELEASES.md entry and the tests a bit please? Once #898 is merged, please also update this branch and keep both sets of tests. |
…s-plan-count test/test_bspot.py conflicted because both branches append after the last pre-existing test. Kept both sets: PythonOT#898's seeding/reproducibility tests and this branch's merge_bijections tests.
merge_bijections_c keeps only the guards that bound the buffer handed to C++ (row length, empty stack); the Python wrapper validates the permutation values. Invalid-input tests are parameterised, and the RELEASES.md entry is shortened.
|
Thanks! I've incorporated the requested cleanup in
#898 is now merged, and this branch is up to date with master (0 commits behind). The current checks are green, including CircleCI Could you please take another look? |
baptiste-genest
left a comment
There was a problem hiding this comment.
Great! Thank you very much!
Problem
ot.bsp.merge_bijections()reads past the end ofpermswhen fewer plans than samples aresupplied, and silently ignores part of its input when more are supplied. Both come from one line:
the number of plans is taken from the wrong axis. Reachable from the documented API:
Reported in #910.
Root cause
permsis a stack of(n_plans, n_samples)permutations — whatcompute_bspot_bijectionreturnsand what the docstring promises.
merge_bijections_cread the plan count from axis 1, the samplecount, and passed it as
nb_plans.MergeBijections()takesnintegers per plan, so it consumesn_samples * n_samplesintegers from a buffer ofn_plans * n_samples:n_plans < n_samplesreads(n_samples - n_plans) * n_samplesintegers out of bounds and feedswhatever it finds to
BijectiveMatchingas permutations;n_plans > n_samplesnever reads plans at index>= n_samples, so they are dropped. Measured ona 7-plan stack over 4 points: the merge returns cost
3.815instead of1.730.bsp_solve_cpassesn_plansexplicitly and is not affected.Fix
ot/bsp/bsp_wrap.pyxreads the plan count fromplans.shape[0].ot/bsp/bsp_ot.pyrejects, before theint32conversion, apermsthat is not 2-D, whose rowsare not one entry per point, that is an empty stack, or whose rows are not a permutation of
0..n_samples-1. The order matters: the cast truncates3.5to3and keeps the low bits of2**32 + 3, either of which would otherwise reach C++ as a valid permutation.merge_bijections_ckeeps only the two guards that bound the buffer it hands to C++ — the rowlength and the empty stack. Per follow-up review, the permutation-value loop is left to the
Python wrapper.
No algorithm change: for a well-formed plan stack the merge receives exactly the plans it was
given.
Two different contracts, deliberately
ot.bsp.merge_bijections(documented API) performs input validation: it rejects apermsthat is not 2-D, has the wrong row length, is empty, or whose rows are not permutations, with
messages naming the expected layout. This is the user-facing guarantee.
merge_bijections_c(internal Cython helper) keeps only two preconditions — row length and anon-empty stack. These are not validation; they are what is needed to bound the buffer handed to
MergeBijections. A well-formed plan stack is a precondition of the helper, and the helper isreached directly only by callers that bypass the public function.
This split is what the follow-up review asked for. The consequence is stated under "Known residual
limitation" rather than left implicit.
Known residual limitation, from the ASan work below
With the native permutation loop removed, calling
merge_bijections_cdirectly with a row that isnot a permutation reaches
BijectiveMatching::getInverseand performs an out-of-bounds write(AddressSanitizer:
WRITE of size 4,0 bytes after 32-byte region,BSP-OT_header_only.h:817, for[0,1,2,3,4,5,6,8]over 8 points;4 bytes beforefor[-1,1,...]).ot.bsp.merge_bijectionscannot reach it — the Python check refuses those rows —but
merge_bijections_cis importable (ot.bsp.bsp_wrap). Flagging it so the choice to rely onthe Python wrapper is made knowingly.
Tests
Seven test functions in
test/test_bspot.py(the invalid-input matrix is parameterised, so the filecollects 30 tests together with #898's).
merge_bijectionshad no direct coverage at all before this; the six pre-existing BSP tests all go through
compute_bspot_bijection, which is why the bug survived, and they pass on the unfixed build.Costs are recomputed in the tests from the returned permutation instead of compared to constants,
so nothing pins a random sequence or a standard library. Reverting only the axis line, with the
guards left in place, makes four tests abort the interpreter and one fail its assertion.
#898 is merged into this branch; both test sets are kept and 30 tests pass.
Memory-safety validation
Ubuntu 24.04 / gcc 13.3,
-fsanitize=address -g -O0 -fopenmp, the realbsp_wrap.pyx+bsp_wrapper.cpp, two variants differing only by the axis line:plans.shape[1](1,32)over 32 pointsheap-buffer-overflow, READ 128, 0 bytes after 128-byte region, viaMergeBijections→bsp_wrapper.cpp:149cost=5.501253128(0,8)over 8 pointsheap-buffer-overflow, READ 32, 0 bytes after 1-byte regionValueError, no report(3,7)over 8 pointsValueErrorfrom the row-length guard(7,4)over 4 pointsWindows/MSVC 14.44 extension:
pytest test/test_bspot.py30 passed;pytest test/test_solvers.py -k bsp6 passed;pytest test/test_solvers.py912 passed,98 skipped.
pre-commit run --all-filesclean for ruff (all three hooks) and codespell; yamllintcomplains about CRLF in workflow YAML on this Windows checkout, unrelated to these files.
Out of scope
compute_bspot_bijection(..., n_plans=0)still aborts the interpreter: it builds its own planstack and never goes through
merge_bijections. That is the separaten_plans=0defect notedunder #898.
Fixes #910