Skip to content

Incorrect format with ExponentialDatumFormatter at resolution limits #165

Open

Description

@jbfaden

Masafumi pointed out the issue when zooming in on an axis. This Jython code shows the root of the problem:

from org.das2.datum.format import ExponentialDatumFormatter
format= ExponentialDatumFormatter(20,7)

print format.format(datum(4.28230E7),Units.dimensionless)  # 0.0922337203685477500E7

Activity

  1. jbfaden commented on Feb 24, 2026

    @jbfaden
    MemberAuthor

    I catch case Masafumi saw, where excessive resolution (20 decimal places)
    resulted in 0.0922337203685477500E7. This is when Math.round(4.282336119000001*(10**20)) results in Long.MAX_VALUE. There is still an issue as to why it thinks it needs so many decimal places.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions