Skip to main content
Version: Current

Data Pipelines - custom components

Persistence Manager​

There is a persistence manager interface available in the genesis-pipeline-api module, which enables you to decide how to manage tracking what you have last persisted.

If you use the default DbPersistenceManager implementation, you can directly inject the interface into your class.

Database Persistence Manager​

The DbPersistenceManager is an implementation of PersistenceManager that stores persistence records in the database as a record of the table mentioned below.

This class supports transactions and uses a transaction handle if provided. Otherwise, it uses the injected instance of the entity database.

The latest index returned will always be the greatest value for the given source. The user is given the option whether or not to ‘append’ the record:

  • if true, create a new persistence record -if false, overwrite the current persistence record with the new index value

As such, there has been a new table created called DATA_PIPELINE_PERSISTENCE which contains the following fields:

  • source: String = the source name for which we are storing the persistence information (needs to be unique per pipeline)
  • index: Long = the index number which will allow us to retrieve the last persisted record

If no other implementation of PersistenceManager has been provided, this implementation will be the default instance of PersistenceManager.