You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Incorrect format with ExponentialDatumFormatter at resolution limits #165
I catch case Masafumi saw, where excessive resolution (20 decimal places)
resulted in 0.0922337203685477500E7. This is when Math.round(4.282336119000001*(10**20)) results in Long.MAX_VALUE. There is still an issue as to why it thinks it needs so many decimal places.
Activity
jbfaden commentedon Feb 24, 2026
I catch case Masafumi saw, where excessive resolution (20 decimal places)
resulted in 0.0922337203685477500E7. This is when Math.round(4.282336119000001*(10**20)) results in Long.MAX_VALUE. There is still an issue as to why it thinks it needs so many decimal places.