Data Pipelines - custom components
Persistence Manager
The persistence manager interface in the genesis-pipeline-api module enables you to specify how to track what you have last persisted.
If you use the default DbPersistenceManager implementation, you can directly inject the interface into your class.
Database Persistence Manager
The DbPersistenceManager is an implementation of PersistenceManager that stores persistence records in the database as a record of the table mentioned below.
This class supports transactions and uses a transaction handle, if provided. Otherwise, it uses the injected instance of the entity database.
The latest index returned will always be the greatest value for the given source. The user is given the option whether or not to ‘append’ the record:
- if true, create a new persistence record
- if false, overwrite the current persistence record with the new index value
The table DATA_PIPELINE_PERSISTENCE contains the following fields for managing persistence:
- source: String = the name of the source for which we are storing the persistence information (this must be unique to each pipeline)
- index: Long = the index number which is used to retrieve the last persisted record
If no other implementation of PersistenceManager
has been provided, this implementation will be the default instance of PersistenceManager.