OpenRate is an open source rating and billing engine for high volume transaction processing, written in Java. It is used to build systems that read usage records (call records, data sessions, events, meter readings), enrich and validate them, apply pricing, and write the results out again.
OpenRate is a framework, not an application. On its own it does nothing useful: you build an application on top of it by configuring a pipeline and extending the abstract modules it provides. See Building an application.
- Version: 1.5.2.6
- Java: 17 (LTS)
- Build: Maven
- License: Apache License 2.0
- Quick start
- Core concepts
- Architecture
- The transaction manager
- Multi-pipeline synchronisation
- Configuration
- Building an application
- Module reference
- The record model
- Rating concepts
- Project layout
- Dependencies
- Development notes and known issues
mvn clean verifyThis compiles, runs the full test suite (155 tests) and produces
target/OpenRate-<version>.jar plus a sources jar.
The tests are self contained: they run against an in-memory HSQLDB instance and need no external database or configuration. They do bind TCP port 8086 on localhost for the ECI, so the suite will fail if that port is already in use.
OpenRate is started by pointing it at a configuration file, which must be on the classpath (not a filesystem path):
java -cp $CLASSPATH OpenRate.OpenRate -p <properties-file.properties.xml>The framework runs in one of two modes, determined by the adapters you configure rather than by a command line switch:
- Batch mode — the pipeline starts, processes whatever input is available, and shuts down. Typical for file and database driven processing.
- Daemon mode — the pipeline starts and runs until stopped, waiting for new work to arrive. Typical for socket, JMS and real time pipelines.
OpenRate.main() returns a non-zero status when startup fails, for example when
the arguments are malformed or the properties file cannot be found on the
classpath. Startup failures print the reason to stdout before exiting.
| Concept | Meaning |
|---|---|
| Framework | The top level process. Owns resources and one or more pipelines. |
| Pipeline | An input adapter, a chain of processing modules, and one or more output adapters. |
| Adapter | Reads records from, or writes records to, the outside world. |
| Processing module (plug in) | A step in the pipeline that inspects or modifies a record. |
| Record | The unit of work travelling through a pipeline. |
| Buffer | The thread safe queue connecting one pipeline stage to the next. |
| Resource | A framework level, shared service: logging, data sources, caches, the transaction manager, the event interface. |
| Cache | Reference data loaded once and shared by processing modules, typically pricing or customer data. |
| Transaction | A batch of work (a file, a database selection) treated as a single commit/rollback unit. |
| Stream | The external unit a transaction wraps, for example one input file. |
A single OpenRate process (the framework) hosts one or more pipelines. Pipelines within a framework share resources and memory, which is the main reason to run several pipelines in one process rather than several processes.
+-----------------------------------+
| OpenRate framework |
| |
Shared resources ----+--> LogFactory |
(one instance, used | DataSourceFactory |
by every pipeline) | CacheFactory |
| TransactionManagerFactory |
| ECI (event control interface) |
| |
| +-----------+ +-----------+ |
| | Pipeline | | Pipeline | .. |
| +-----------+ +-----------+ |
+-----------------------------------+
Each pipeline is a linear chain. Stages are connected by buffers, and each stage runs in its own thread inside a dedicated thread group:
external +--------+ +--------+ +--------+ external
source --------> | Input | -buffer-> | Proc 1 | -buffer-> | Proc N | -buffer-> | Output | --------> sink
| Adapter | | module | | module | | Adapter|
+--------+ +--------+ +--------+ +--------+
Key properties:
- Records move in batches. Each module has a configurable
BatchSizecontrolling how many records it takes from its input buffer per cycle. This amortises the cost of thread handoff; record-at-a-time processing would be far slower. - Stages run concurrently. While module N processes batch 1, module N-1 can already be processing batch 2.
- Output adapter chaining is supported, so one record can be written to several outputs. If an output is not explicitly named, the record goes to all of them.
- The pipeline is assembled reflectively from configuration in
Pipeline.ConfigurePipeline(). Modules are named in the config file by their fully qualified class name.
Streams are delimited by control records rather than by adapter callbacks:
- A HeaderRecord is emitted when a new stream (for example a file) starts. This opens a transaction.
- A TrailerRecord is emitted when the stream ends. This begins transaction close down.
Every processing module sees these records and may act on them — this is how modules know to flush per-stream state. Modules that do not care simply pass them through.
OpenRate deliberately does not use record level (J2EE style) transactions,
which are far too costly for the volumes involved. Instead, each stream is
wrapped in a transaction, managed by OpenRate.transaction.TransactionManager.
Every module that participates registers as a client of the transaction. The transaction advances only when all clients agree, and its overall status is the lowest status reported by any client.
Defined in OpenRate.transaction.TMDefs:
| Value | State | Meaning |
|---|---|---|
| 0 | TM_NONE |
No transaction. |
| 1 | TM_PROCESSING |
Records are flowing. |
| 2 | TM_FLUSHED |
All records have passed through; clients may now commit. |
| 3 | TM_FINISHED_ERR |
Client finished, but with errors. |
| 4 | TM_FINISHED_OK |
Client finished cleanly. |
| 5 | TM_CLOSING |
Transaction is closing down. |
| 6 | TM_CLOSED |
Complete. |
If any client reports TM_FINISHED_ERR, or an abort is requested, the whole
transaction is marked errored and rolls back.
The TransactionFlusher runs as a separate thread per pipeline. Once every
client has flushed, the transaction is queued to the flusher, which drives it
through commit or rollback and then closes it.
Some operations — notably reloading a shared cache — cannot safely run while pipelines are processing. The framework coordinates this with a sync protocol across all pipelines in the process:
- A pipeline that needs a sync point sets its sync status to 1 (
SyncFlag). - The framework sees the flag and sets every pipeline's status to 2
(
SyncRequest). - Each pipeline must stop after finishing its current transaction, then report
3 (
SyncReached). - Once all pipelines report
SyncReached, the framework sets them to 4 (SyncProcess). Pending events are processed now. - Each pipeline reports 5 (
SyncDone) when finished. - When all are
SyncDone, the framework resets everyone to 0 and normal processing resumes.
This is why a cache reload in one pipeline briefly pauses the others.
A single XML file configures the entire framework. It must be on the classpath.
Note that despite the .properties.xml extension it is ordinary XML, not a Java
properties file.
<config>
<Application> ... </Application>
<!-- which pipelines exist, and whether they start -->
<PipelineList>
<MyPipe><Active>True</Active></MyPipe>
</PipelineList>
<!-- one block per pipeline, named to match PipelineList -->
<MyPipe>
<InputAdapter>
<MyInput>
<ClassName>OpenRate.adapter.file.FlatFileInputAdapter</ClassName>
<BatchSize>5000</BatchSize>
</MyInput>
</InputAdapter>
<Process>
<MyLookupStep>
<ClassName>com.example.MyLookupPlugIn</ClassName>
<DataCache>MyCache</DataCache>
<BatchSize>5000</BatchSize>
</MyLookupStep>
</Process>
<OutputAdapter>
<MyOutput>
<ClassName>OpenRate.adapter.file.FlatFileOutputAdapter</ClassName>
<BatchSize>5000</BatchSize>
</MyOutput>
</OutputAdapter>
</MyPipe>
<!-- framework wide shared services -->
<Resource>
<LogFactory> ... </LogFactory>
<ECI> ... </ECI>
<TransactionManagerFactory> ... </TransactionManagerFactory>
<DataSourceFactory> ... </DataSourceFactory>
<CacheFactory> ... </CacheFactory>
</Resource>
</config>Processing modules execute in the order they appear inside <Process>.
LogFactory — wires up logging. Properties names a Log4j2 configuration
file, also resolved from the classpath.
<LogFactory>
<ClassName>OpenRate.logging.LogFactory</ClassName>
<Properties>log4j2.xml</Properties>
<DefaultCategory>MyApp</DefaultCategory>
</LogFactory>DataSourceFactory — defines JDBC connection pools. DataSourceBuilder
selects the pool implementation; each child of <DataSource> is a named pool
referenced elsewhere by that name.
<DataSourceFactory>
<ClassName>OpenRate.resource.DataSourceFactory</ClassName>
<DataSourceBuilder>
<ClassName>OpenRate.db.C3P0DataSource</ClassName>
</DataSourceBuilder>
<DataSource>
<MyDB>
<db_url>jdbc:hsqldb:mem://localhost/testdb</db_url>
<driver>org.hsqldb.jdbc.JDBCDriver</driver>
<username>SA</username>
<password></password>
<ValidationQuery>SELECT 1 FROM INFORMATION_SCHEMA.SYSTEM_USERS</ValidationQuery>
<TestConnectionPeriod>60</TestConnectionPeriod>
</MyDB>
</DataSource>
</DataSourceFactory>Available pool builders: OpenRate.db.C3P0DataSource,
OpenRate.db.HikariCPDataSource.
CacheFactory — declares reference data caches. Each cache names its
implementation class, where its data comes from (DB or File), and the query
or file that loads it.
<CacheFactory>
<ClassName>OpenRate.resource.CacheFactory</ClassName>
<CacheableClass>
<MyCache>
<ClassName>OpenRate.cache.BestMatchCache</ClassName>
<DataSourceType>DB</DataSourceType>
<DataSource>MyDB</DataSource>
<SelectStatement>select MAP_GROUP, INPUT_VAL, RESULT from MY_MAP</SelectStatement>
</MyCache>
</CacheableClass>
</CacheFactory>Processing modules bind to a cache with <DataCache>MyCache</DataCache>.
ECI (Event Control Interface) — opens an admin socket for querying and controlling a running framework: inspecting module state, triggering cache reloads, and shutting down cleanly.
<ECI>
<ClassName>OpenRate.configurationmanager.EventHandler</ClassName>
<Port>8086</Port>
<MaxConnection>2</MaxConnection>
</ECI>Connect with any line oriented client (telnet, nc). Modules expose their own
commands by registering services through ClientManager.
Security note: the ECI port is an unauthenticated control channel. Bind it to a trusted interface and firewall it.
-
Depend on the OpenRate jar. OpenRate is not published to Maven Central, so build and install it into your local repository first:
git clone https://github.com/isparkes/OpenRate.git cd OpenRate && mvn clean install
Then depend on it:
<dependency> <groupId>OpenRate</groupId> <artifactId>OpenRate</artifactId> <version>1.5.2.6</version> </dependency>
-
Define a record type. Extend
OpenRate.record.AbstractRecord, or useFlatRecord/RatingRecorddirectly. A record normally implements:mapOriginalData()— raw input to structured fields (mandatory)unmapOriginalData()— structured fields back to flat output (needed to write files)getDumpInfo()— human readable dump for diagnostics (optional)
-
Extend the processing modules you need. Most classes in
OpenRate.processare namedAbstract*because they implement the mechanics and leave the business decision to you. A typical plug in extendsAbstractPlugIn(orAbstractTransactionalPlugIn) and implements:procValidRecord(IRecord r)— the normal pathprocErrorRecord(IRecord r)— records already flagged with errorsprocHeader(...)/procTrailer(...)— stream boundaries
-
Pick adapters, or extend one if your format is unusual.
-
Write the configuration file and put it on the classpath.
-
Run it.
Complete worked pipelines live in separate repositories under
github.com/isparkes. All of them pin older
OpenRate versions, so each needs its dependency bumped to 1.5.2.6 and its
compiler set to <release>17</release>.
Each was migrated and built against OpenRate 1.5.2.6 on Java 17 to confirm the steps below are sufficient:
| Application | What it demonstrates | Pins | Work needed |
|---|---|---|---|
| OpenRateSimple | Simplest non-trivial pipeline — start here | 1.5.2.4 | pom only |
| ORPixip | Revenue assurance over five sources, re-rating and comparison | 1.5.2.4 | pom + config |
| OpenRateWeblinksInterconnect | Interconnect rating | 1.5.2.5 | pom + config |
| ORTyfon | Fixed line and VoIP reseller rating, fraud detection | 1.5.2.2 | pom + config + API (21 files) |
| ORSeaSatCom | GSM and satellite reseller retail rating | 1.5.2.2 | pom + config + API (16 files) |
ORTyfon and ORSeaSatCom pin 1.5.2.2 and predate API changes made between
1.5.2.2 and 1.5.2.6. Nothing recent broke them. The required changes are
mechanical:
1. Record fields moved to camel case. These are public fields on
RatingRecord and AbstractRecord:
| Was | Now |
|---|---|
OriginalData |
originalData |
UTCEventDate |
utcEventDate |
EventStartDate / EventEndDate |
eventStartDate / eventEndDate |
RecordNumber |
recordNumber |
CounterCycle |
counterCycle |
CreateBreakdown |
createBreakdown |
Service |
service |
Note that FlatRecord.originalData is private — subclasses of FlatRecord
must use getData() / setData() instead.
2. Adapter signatures are strongly typed. The base classes now declare:
// FlatFileInputAdapter
public abstract IRecord procValidRecord(FlatRecord r);
public abstract IRecord procErrorRecord(FlatRecord r);
// FlatFileOutputAdapter
public abstract Collection<FlatRecord> procValidRecord(IRecord r);
public abstract Collection<FlatRecord> procErrorRecord(IRecord r);
// JDBCOutputAdapter - same shape, with DBRecord
public abstract Collection<DBRecord> procValidRecord(IRecord r);
// every adapter
public abstract HeaderRecord procHeader(HeaderRecord r);
public abstract TrailerRecord procTrailer(TrailerRecord r);Knock-on changes in the method bodies:
- local
Collection<IRecord>batches becomeCollection<FlatRecord>orCollection<DBRecord>to match the return type outbatch.add((IRecord) rec)casts are no longer valid — drop the castreturn (IRecord) trailer;no longer compiles where the return type isTrailerRecord— drop the cast- add imports for
HeaderRecord,TrailerRecord,FlatRecordorDBRecordas the new signatures require
Check for partially migrated code before converting: WIInputAdapter in
OpenRateWeblinksInterconnect already carries the typed methods alongside the
old IRecord overloads, and converting those overloads produces duplicate
methods rather than a fix.
3. The MySQL driver class was renamed. In the .properties.xml files,
com.mysql.jdbc.Driver becomes com.mysql.cj.jdbc.Driver. Four of the five
examples need this.
Other OpenRate related repositories, not build tested: ORFuzer, OpenRateProv, OpenRateProvWS.
| Package | Adapters |
|---|---|
adapter.file |
FlatFileInputAdapter, FlatFileNTInputAdapter, BinaryFileInputAdapter, XMLFileInputAdapter, ASN1FileInputAdapter |
adapter.jdbc |
JDBCInputAdapter |
adapter.jms |
AbstractAMQInputAdapter (ActiveMQ) |
adapter.socket |
SocketInputAdapter |
adapter.objectInterface |
ObjectInputAdapter, BufferInputAdapter |
adapter.listener |
ListenerNTInputAdapter |
adapter |
NullInputAdapter (testing) |
File input adapters select files using a glob of the form
<prefix>*<suffix>, where * must match at least one character.
| Package | Adapters |
|---|---|
adapter.file |
FlatFileOutputAdapter, FlatFileNTOutputAdapter, FlatFileMultiStreamOutputAdapter |
adapter.jdbc |
JDBCOutputAdapter, JDBCBatchOutputAdapter, MySQLDirectLoadOutputAdapter, PgSQLCopyOutputAdapter |
adapter.jms |
AbstractAMQOutputAdapter |
adapter.socket |
SocketOutputAdapter |
adapter |
NullOutputAdapter, NullNTOutputAdapter |
MySQLDirectLoadOutputAdapter and PgSQLCopyOutputAdapter use native bulk load
paths and are dramatically faster than row-by-row inserts for large volumes.
adapter.realTime provides a request/response path that bypasses batch
buffering for latency sensitive work (for example online credit checks):
AbstractRTAdapter, AbstractRTSocketAdapter, AbstractRTMethodAdapter,
RTSocketListener, RTMethodListener, RTProcessingProxy.
AbstractTeeAdapter, AbstractBufferTeeAdapter and AbstractSocketTeeAdapter
duplicate a record stream to a secondary destination without disturbing the main
flow — useful for auditing and monitoring.
Lookup and matching
AbstractBestMatch, AbstractBestMatchFixedLine, AbstractRegexMatch,
AbstractIndexedLookupMatch, AbstractNumberRangeMatch,
AbstractMultipleValidityMatch, AbstractValidityFromLookup,
AbstractValiditySegmentMatch, AbstractTimeMatch
Rating
AbstractRateCalc, AbstractRUMRateCalc, AbstractRUMCPRateCalc,
AbstractRUMMap, AbstractRUMBestMatch, AbstractRUMBestMatchFixedLine,
AbstractRUMTimeMatch, AbstractRUMHolidayMatch, AbstractMinFeeCalc,
AbstractProRation, GatherRUMImpacts
Customer and balance
AbstractCustomerLookup, AbstractCustomerLookupAudited,
AbstractJBCustomerLookup, AbstractBalanceHandlerPlugIn
Control and utility
AbstractPlugIn, AbstractTransactionalPlugIn, AbstractStubPlugIn,
AbstractDuplicateCheck, AbstractAggregation, AbstractCallAssembly,
AbstractPersistentObjectProcess, AbstractTransientHash,
AbstractRecordInsertPlugIn, AbstractEmailNotification, ProcessStateLogger,
Dump, DumpNT, DumpRT
Reference data is loaded once into a cache and shared across modules. Caches
implement ICacheable and are loaded by ICacheLoader; those implementing
ICacheAutoReloadable can be reloaded at runtime via the ECI (which triggers
the sync protocol described above).
BestMatchCache, BestMatchFixedLineCache, RegexMatchCache,
IndexedLookupCache, NumberRangeCache, MultipleValidityCache,
ValidityFromCache, ValiditySegmentCache, TimeModelCache, RateCache,
RUMRateCache, RUMCPRateCache, RUMMapCache, RUMHolidayMatchCache,
MinFeeCache, BalanceCache, CustomerCache, CustomerCacheAudited,
JBCustomerCache, AggregationCache, CallAssemblyCache,
DuplicateCheckCache, PersistentIndexedObject, TransientHash
ArrayListQueueBuffer and LinkedListQueueBuffer implement IBuffer and
connect pipeline stages. LinkedListQueueBuffer is backed by
ConcurrentLinkedDeque.
All records derive from AbstractRecord and implement IRecord.
| Record | Purpose |
|---|---|
FlatRecord |
A raw line of text, as read or written by a file adapter. |
RatingRecord |
The main structured record for rating, carrying charge packets and RUMs. |
DBRecord |
Database sourced record. |
HeaderRecord / TrailerRecord |
Stream boundary markers (see above). |
KeyValuePairRecord |
Generic key/value payload. |
QueueMessageRecord |
JMS message wrapper. |
Records accumulate errors rather than throwing: a module attaches a
RecordError (see IError, ErrorType, AbstractError) and the record
continues down the pipeline, where later modules and the output adapter can
route it to an error sink. This keeps one bad record from stopping a run.
RUM — rateable Usage Metric. A named, quantified unit on a record that pricing is applied to: seconds of call duration, megabytes transferred, number of events. A single record may carry several RUMs, each priced differently.
Charge packet (ChargePacket) — holds the rating information driving zoning
and rating. Each time the applicable rating changes, a new charge packet is
created, so one record may hold many.
Price model — a named set of tiers and beats. Rating types supported by the RUM map include:
Flat— simpleRUM × price, no tiers or beatsTiered— price varies by consumption band- and further models described in the
RUMMapCachejavadoc
Beat — the granularity of charging; for example 60 second beats round a 45 second call up to a full minute.
Resource — in a rating context, the thing a charge impacts: a currency, a
bundle allowance, a counter. BalanceImpact records the effect on a balance.
RUM map — maps price groups to the price models, RUMs and resources they
affect. Loaded by RUMMapCache from file or database.
src/main/java/OpenRate/
OpenRate.java framework entry point and lifecycle
Pipeline.java pipeline assembly and execution
IPipeline.java pipeline contract
CommonConfig.java shared configuration constants
adapter/ input and output adapters (file, jdbc, jms,
socket, realTime, objectInterface, listener)
buffer/ inter-stage queues
cache/ reference data caches
configurationmanager/ ECI: event handling and admin sockets
db/ JDBC helpers and connection pool builders
exception/ exception hierarchy
lang/ domain types (customer, product, balance helpers)
logging/ logging abstraction over Log4j2
parser/ parsers, including ASN.1
process/ processing modules
record/ record model
resource/ framework level shared resources
threads/ thread pool
transaction/ transaction manager and flusher
utils/ conversion and general utilities
src/test/
java/OpenRate/ unit and integration tests
java/TestUtils/ framework bootstrap helpers for tests
resources/ test configurations (*.properties.xml)
Runtime dependencies are deliberately minimal:
| Dependency | Purpose |
|---|---|
log4j-api / log4j-core |
Logging |
c3p0 |
JDBC connection pool |
HikariCP |
JDBC connection pool (alternative) |
activemq-client |
JMS adapters |
angus-mail |
Email notification module |
mysql-connector-j |
MySQL driver |
postgresql |
PostgreSQL driver |
hsqldb |
HSQLDB driver, and the test database |
junit |
Tests only |
The JDBC drivers are provided for convenience; a deployment only needs the one matching its database.
The suite is timing sensitive in places because it exercises concurrent
transaction handling. It is also timezone and locale independent — verified
under TZ=UTC as well as CET.
The suite binds localhost port 8086 (the ECI), so it cannot run twice concurrently on one machine.
Always run the suite with a timeout, so a concurrency regression fails the build instead of hanging it:
mvn -B clean verify -Dsurefire.timeout=400C3P0DataSource sets both maxStatements and maxStatementsPerConnection to
0. This is deliberate. c3p0 documents these as alternatives and warns against
setting both; from c3p0 0.10 onward, setting both causes cached statements
belonging to connections already returned to the pool to be handed out, which
surfaces as the SQL error connection exception: closed once transactions
overlap.
Override per data source with MaxStatements / MaxStatementsPerConnection
only if you
have verified the behaviour with your driver. Most modern drivers cache server
side anyway.
Some tests in AbstractDuplicateCheckTest open a transaction and close it at
the end of the test body. If an assertion fails in between, the transaction is
leaked, and any later test that waits for all transactions to close will hang
rather than fail. When a test in this class hangs, look for an earlier test
failure as the real cause.
ThreadGroup.destroy() is called in OpenRate.java, Pipeline.java and
CacheFactory.java. It is deprecated for removal and throws
UnsupportedOperationException on Java 21+. This must be addressed before
moving beyond Java 17. The CI workflow runs Java 21 and 25 as advisory
(non-blocking) jobs to track this.
.github/workflows/build.yml gates on Java 17 and additionally runs advisory
builds on newer JDKs. Test reports are uploaded on failure.
The code is the primary documentation, and the javadoc is substantial —
particularly on OpenRate, Pipeline, TransactionManager and the
Abstract* process modules, which document their configuration properties
inline. Generate it with:
mvn javadoc:javadoc -Ddoclint=noneOutput lands in target/reports/apidocs/. The -Ddoclint=none is required:
a number of javadoc comments embed unescaped XML and HTML examples, which
JDK 17's doclint rejects as malformed. Without the flag the build fails.
Each package also carries a package.html describing its role.
Licensed under the Apache License, Version 2.0. See LICENSE.
Copyright 2006-2026 Ian Sparkes.