Skip to content

Fix the loss of precision of von_mises_lpdf and its κ-derivatives for large κ - #3512

Open
jachymb wants to merge 1 commit into
stan-dev:developfrom
jachymb:bugfix/von-mises-lpdf-cancellation
Open

jachymb wants to merge 1 commit into
stan-dev:developfrom
jachymb:bugfix/von-mises-lpdf-cancellation

Conversation

@jachymb

@jachymb jachymb commented Nov 13, 2026

Copy link
Copy Markdown
Contributor

AI use disclosure: This was developed with the help of Claude Code.

Summary

Fixes the documented numerical stability problem of von Mises distribution: https://mc-stan.org/docs/functions-reference/circular_distributions.html#numerical-stability

von_mises_lpdf forms its value and its κ-partial as differences of terms that grow with κ:

$$\log p = \kappa \cos(y - \mu) - \log 2\pi - \log I_0(\kappa), \qquad \partial_\kappa \log p = \cos(y - \mu) - \exp(\log I_1(\kappa) - \log I_0(\kappa)).$$

Where the density is concentrated, $y - \mu$ is of order $\kappa^{-1/2}$. The value then subtracts two terms of size κ. The κ-partial subtracts two numbers close to 1, one of them the exponential of a difference of two logs of size κ. Its derivatives, which nested autodiff takes through log_modified_bessel_first_kind, cancel the same way. So the value has an absolute error of about κε, and the κ-derivatives relative errors of up to about κ²ε. From κ ≈ 10⁸ on, $\log I_1 - \log I_0$ (about $-\frac{1}{2\kappa}$ ) is below the rounding error of the two logs, so the computed ratio is exactly 1 and the κ-gradient loses its $1/(2\kappa)$ term, a third of the gradient at $y - \mu = 2/\sqrt{\kappa}$. The Functions Reference warns that the distribution is numerically unstable for κ > 100.

The lpdf now uses the exactly equivalent form, from $\kappa \cos d - \kappa = -2\kappa \sin^2(d/2)$:

$$\log p = -2\kappa \sin^2\frac{y - \mu}{2} - \log 2\pi - \log g(\kappa), \qquad \partial_\kappa \log p = \rho_0(\kappa) - 2 \sin^2\frac{y - \mu}{2},$$

with $g(z) = e^{-z} I_0(z)$ and $\rho_0 = 1 - I_1/I_0 = -g'/g$. The y- and μ-partials are unchanged.

Both Bessel terms come from two new functions in prim/fun (C++ only, no Stan signature), which the cdf (suffering a similar problem) can later reuse. They rest on two facts about $g$: it is $\frac{1}{\pi} \int_0^\pi e^{-z (1 - \cos\theta)} d\theta$, so its derivatives alternate in sign, and it satisfies $z g'' + (2z + 1) g' + g = 0$.

  • bessel_i0e_derivative_ratio(n, z) $= \rho_n(z) = -g^{(n+1)}(z) / g^{(n)}(z)$. Differentiating the equation $n$ times gives a continued fraction, and the definition gives the derivative:
    $$\rho_n = \frac{2n + 1}{n + 1 + 2z - z \rho_{n+1}}, \qquad \rho_n' = \rho_n (\rho_n - \rho_{n+1}).$$
    Every $\rho_n$ lies in (0, 2), so the denominators are at least $n + 1$: the fraction gives the complement $1 - I_1/I_0$ itself, with no term of size $z$ or 1 to cancel. In double it is summed by the three-term recurrence of its convergents. It needs up to 59 terms for $z$ between 5 and 20, so for $z &lt; 25$ the function forms $\rho_0 = (I_0 - I_1)/I_0$ from the Boost approximations of $I_0$ and $I_1$ that log_modified_bessel_first_kind already uses: there the subtraction loses less than two digits (relative error below 3e-14, against 3e-15 for the fraction). For autodiff types the second relation is the partial, so the derivatives of every order come from values of the same function and nothing is differentiated through an iteration: one call puts 2 nodes on the tape for a var argument and 8 for fvar<var>.
  • log_bessel_i0e(z) $= \log g(z)$, with derivative $-\rho_0$. Its value comes from the same approximation of $I_0$; for $z \ge 7.75$ that is a fit of $\sqrt{z} e^{-z} I_0(z)$, so $z$ is never added and subtracted.

For the two functions, three coefficient tables of $I_0$ and two of $I_1$ move from function scope to namespace scope in log_modified_bessel_first_kind.hpp; that function stays bit-identical to develop (values and gradients, 12,003 inputs).

Relative errors at $y - \mu = 2/\sqrt{\kappa}$ against mpmath (60 digits), develop → this PR:

κ value ∂κ ∂²κ ∂³κ
10⁴ 7.9e-13 → 3.3e-16 2.5e-9 → 1.5e-16 4.6e-9 → 1.6e-16 1.5e-11 → 3.9e-16
10⁶ 1.7e-11 → 1.7e-16 2.4e-6 → 2.5e-16 3.3e-5 → 1.0e-16 9.5e-11 → 9.2e-19
10⁸ 6.4e-10 → 1.3e-16 0.33 → 2.0e-16 0.14 → 1.2e-16 5.9e-9 → 2.3e-16
10¹² 3.2e-6 → 1.5e-17 0.33 → 4.1e-17 8.9e7 → 2.4e-16 3.3e-6 → 4.9e-16

At 22 values of κ from 0.5 to 10¹⁵ the largest errors are now 3.2e-15 (value), 4.1e-15 (∂κ), 5.7e-15 (∂²κ) and 4.5e-15 (∂³κ), the last three at κ = 20; up to κ = 30 develop reached 5.4e-14 (∂κ), 1.8e-13 (∂²κ) and 1.4e-11 (∂³κ). The y- and μ-derivatives were already accurate and are bit-identical to develop.

The third order matters for the embedded Laplace approximation. With a latent log κ (six observations, squared-exponential kernel), the Laplace marginal density and its gradient in the kernel parameters agree with a likelihood written from the large-κ series to 3e-14 and 2e-11 at κ = 10⁶, 10⁹ and 10¹²; on develop the gradient is off by 3e-5 at 10⁶, and by 25% and 430% in its two components at 10⁹.

On Linux the usual vectorized call runs at develop's speed (Side Effects), and calls that evaluate the Bessel terms per observation are faster: 0.43–0.78 of develop's time for all-scalar calls, 0.60–0.65 for a vector κ, and 0.17 with 6% of the arena memory for fvar<var> with a vector κ. The lpdf header and its mix test also drop unused includes.

Not changed: von_mises_cdf, von_mises_lcdf, von_mises_lccdf and von_mises_rng have their own large-κ problems (I can do that after this gets merged). The Functions Reference note needs a separate docs PR once this is merged.

Tests

  • mix/prob/von_mises_test: the new mathMixScalFun_von_mises_lpdf_large_kappa compares, at κ = 10¹², the value, the κ-gradient (plain var, the partial in fvar<fvar<var>>, and the derivative of the value itself), the second and the third κ-derivative with mpmath, to 1e-13 relative. Hand-written fvar<fvar<var>> checks are used because finite differences cannot resolve these derivatives at κ = 10¹². On develop it fails: the value is 10.89661 instead of 10.89657, the gradient −2.0e-12 instead of −1.5e-12 (the value's own derivative −1.49991e-12), the second derivative −0 instead of −5.0e-25, and the third 9.99997e-37 instead of 1.00000e-36.
  • prim/fun/log_bessel_i0e_test and prim/fun/bessel_i0e_derivative_ratio_test: values against mpmath at points covering $z = 0$, the switches of the evaluations (7.75 and 500 for $I_0$; 7.75 and 25 for the ratio, and 21, where its error is largest), 10¹², 10³⁰⁰, 1.7e308 (ratio) and ∞; orders 0, 1, 3 and the largest order, 1000; integer and vector arguments; the error paths.
  • mix/fun/log_bessel_i0e_test and mix/fun/bessel_i0e_derivative_ratio_test: expect_ad (first to third order against finite differences, and a throw in every autodiff type for a negative argument) at moderate $z$, for orders 0 and 2, and third-order checks against mpmath at $z$ = 0.5, 15, 10⁶ and 10¹².
  • Existing tests that reach the code pass: the expect_ad test of the lpdf, the generated test/prob/von_mises lpdf tests (clang++ 18, O=0, N_TESTS=100), prim/prob/von_mises_cdf_test, the prim and mix tests of log_modified_bessel_first_kind, the three rev integrate_1d tests that integrate the density, and the expression tests of von_mises_lpdf and log_modified_bessel_first_kind (clang++ 18).

Side Effects

On Linux the usual call, vectorized over observations with one κ parameter, runs at develop's speed (1.02–1.03), and the only slower path is nested autodiff with a vector μ (1.11). On Windows with MinGW g++ 15 the usual call takes 1.3 times as long, from an inlining-budget decision of GCC (1.01 with a larger budget). Every other measured path is faster than develop on both platforms, down to 0.17 of its time.

Paths more than 5% slower on either platform, relative to develop (one process, -O3 -DNDEBUG):

path WSL, g++ 14 MinGW, g++ 15
vector μ, scalar κ (usual) 1.02–1.03 1.33–1.35
vector y, data κ 1.03 1.16
fvar<var>, vector μ 1.11 1.15

Arena memory: unchanged at first order, 1.3 times for fvar<var> with a vector μ. For κ < 30, values and κ-gradients are within 1.3e-13 of develop.

Behavior change: with propto = true and data κ, the result changes by the constant $-\kappa$ per term.

Release notes

von_mises_lpdf and its derivatives are now accurate for large concentration κ.

Checklist

  • Copyright holder: Jáchym Barvínek jachymb@gmail.com

    The copyright holder is typically you or your assignee, such as a university or company. By submitting this pull request, the copyright holder is agreeing to the license the submitted work under the following licenses:
    - Code: BSD 3-clause (https://opensource.org/licenses/BSD-3-Clause)
    - Documentation: CC-BY 4.0 (https://creativecommons.org/licenses/by/4.0/)

  • the basic tests are passing

    • unit tests pass (to run, use: ./runTests.py test/unit)
    • header checks pass, (make test-headers)
    • dependencies checks pass, (make test-math-dependencies)
    • docs build, (make doxygen)
    • code passes the built in C++ standards checks (make cpplint)
  • the code is written in idiomatic C++ and changes are documented in the doxygen

  • the new changes are tested

The value and the kappa-derivatives subtracted terms that grow with
kappa. Compute the lpdf as -2 kappa sin^2((y - mu)/2) - log(2 pi)
- log_bessel_i0e(kappa) with kappa-partial
bessel_i0e_derivative_ratio(0, kappa) - 2 sin^2((y - mu)/2).

log_bessel_i0e returns log I0(z) - z from Boost's I0 approximation, with
the exact derivative -(1 - I1/I0).

bessel_i0e_derivative_ratio(n, z) returns the ratio of successive
derivatives of exp(-z) I0(z), which is 1 - I1/I0 for n = 0. It comes from
Boost's I0 and I1 approximations where their difference keeps its
precision (n = 0, z < 25) and otherwise from a continued fraction for the
complement itself. Its derivative is rho_n (rho_n - rho_{n+1}), so nested
autodiff gets second and third derivatives from values of the same
function.

The I0 and I1 coefficient tables move to namespace scope in
log_modified_bessel_first_kind.hpp; that function stays bit-identical.
@stan-buildbot

Copy link
Copy Markdown
Contributor
Name Old Result New Result Ratio Performance change( 1 - new / old )
stat_comp_benchmarks/benchmarks/gp_regr/gp_regr.stan 0.25 0.25 0.99 -1.25% slower
stat_comp_benchmarks/benchmarks/gp_regr/gen_gp_data.stan 0.06 0.06 0.99 -0.97% slower
stat_comp_benchmarks/benchmarks/low_dim_gauss_mix/low_dim_gauss_mix.stan 6.6 6.59 1.0 0.12% faster
stat_comp_benchmarks/benchmarks/low_dim_corr_gauss/low_dim_corr_gauss.stan 0.02 0.02 1.01 1.41% faster
stat_comp_benchmarks/benchmarks/irt_2pl/irt_2pl.stan 9.37 9.32 1.0 0.48% faster
stat_comp_benchmarks/benchmarks/gp_pois_regr/gp_pois_regr.stan 4.67 4.66 1.0 0.34% faster
stat_comp_benchmarks/benchmarks/sir/sir.stan 225.84 224.78 1.0 0.47% faster
stat_comp_benchmarks/benchmarks/garch/garch.stan 1.01 1.03 0.98 -2.53% slower
stat_comp_benchmarks/benchmarks/arma/arma.stan 0.69 0.7 1.0 -0.5% slower
stat_comp_benchmarks/benchmarks/pkpd/one_comp_mm_elim_abs.stan 43.23 42.78 1.01 1.04% faster
stat_comp_benchmarks/benchmarks/pkpd/sim_one_comp_mm_elim_abs.stan 0.58 0.59 0.99 -1.5% slower
stat_comp_benchmarks/benchmarks/low_dim_gauss_mix_collapse/low_dim_gauss_mix_collapse.stan 21.74 21.72 1.0 0.1% faster
stat_comp_benchmarks/benchmarks/eight_schools/eight_schools.stan 0.11 0.11 1.01 0.85% faster
stat_comp_benchmarks/benchmarks/arK/arK.stan 3.4 3.41 1.0 -0.03% slower
performance.compilation 377.13 373.79 1.01 0.89% faster
Mean result: 0.9993864438406682

Jenkins Console Log
Jenkins Build Stages
Commit hash: 391d3a4b1515dc036b8b287d10029ea398102ec2

Machine information
Distributor ID:	Ubuntu
Description:	Ubuntu 20.04.3 LTS
Release:	20.04
Codename:	focal

CPU:

Architecture:                            x86_64
CPU op-mode(s):                          32-bit, 64-bit
Byte Order:                              Little Endian
Address sizes:                           43 bits physical, 48 bits virtual
CPU(s):                                  256
On-line CPU(s) list:                     0-255
Thread(s) per core:                      2
Core(s) per socket:                      64
Socket(s):                               2
NUMA node(s):                            2
Vendor ID:                               AuthenticAMD
CPU family:                              23
Model:                                   49
Model name:                              AMD EPYC 7742 64-Core Processor
Stepping:                                0
Frequency boost:                         enabled
CPU MHz:                                 1496.365
CPU max MHz:                             3416.0681
CPU min MHz:                             1500.0000
BogoMIPS:                                4491.80
Virtualization:                          AMD-V
L1d cache:                               4 MiB
L1i cache:                               4 MiB
L2 cache:                                64 MiB
L3 cache:                                512 MiB
NUMA node0 CPU(s):                       0-63,128-191
NUMA node1 CPU(s):                       64-127,192-255
Vulnerability Gather data sampling:      Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit:             Not affected
Vulnerability L1tf:                      Not affected
Vulnerability Mds:                       Not affected
Vulnerability Meltdown:                  Not affected
Vulnerability Mmio stale data:           Not affected
Vulnerability Old microcode:             Not affected
Vulnerability Reg file data sampling:    Not affected
Vulnerability Retbleed:                  Mitigation; untrained return thunk; SMT enabled with STIBP protection
Vulnerability Spec rstack overflow:      Mitigation; Safe RET
Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1:                Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Vulnerability Spectre v2:                Mitigation; Retpolines; IBPB conditional; STIBP always-on; RSB filling; PBRSB-eIBRS Not affected; BHI Not affected
Vulnerability Srbds:                     Not affected
Vulnerability Tsa:                       Not affected
Vulnerability Tsx async abort:           Not affected
Vulnerability Vmscape:                   Mitigation; IBPB before exit to userspace
Flags:                                   fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl xtopology nonstop_tsc cpuid extd_apicid aperfmperf rapl pni pclmulqdq monitor ssse3 fma cx16 sse4_1 sse4_2 x2apic movbe popcnt aes xsave avx f16c rdrand lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw ibs skinit wdt tce topoext perfctr_core perfctr_nb bpext perfctr_llc mwaitx cpb cat_l3 cdp_l3 hw_pstate ssbd mba ibrs ibpb stibp vmmcall fsgsbase bmi1 avx2 smep bmi2 cqm rdt_a rdseed adx smap clflushopt clwb sha_ni xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local clzero irperf xsaveerptr rdpru wbnoinvd amd_ppin arat npt lbrv svm_lock nrip_save tsc_scale vmcb_clean flushbyasid decodeassists pausefilter pfthreshold avic v_vmsave_vmload vgif v_spec_ctrl umip rdpid overflow_recov succor smca sev sev_es

G++:

g++ (Ubuntu 9.4.0-1ubuntu1~20.04) 9.4.0
Copyright (C) 2019 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

Clang:

clang version 10.0.0-4ubuntu1 
Target: x86_64-pc-linux-gnu
Thread model: posix
InstalledDir: /usr/bin

@avehtari avehtari added numerics Numerical issues distributions Issues that deal with distribution functions: pdf, pmf, cdf labels Nov 15, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

distributions Issues that deal with distribution functions: pdf, pmf, cdf numerics Numerical issues

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants