The way businesses operate is continuously changing and as the world is becoming a global village, importance of data and the size of the data in any organization is growing rapidly every moment as we speak. For Mid to Large IT organizations, the old way of consuming data once or twice a day from the source system will no longer work. With businesses expending across the globe, it can be a difficult process for senior leadership to get quick insights for their operations, especially if transactions are not available to the next day. Streaming data provides the necessary speed which gives flexibility for faster insight, often with a single dashboard click for performance checks.

While building and configuring the data pipelines for supporting data streaming, choosing the right replication method is the most important factor. Incorrectly choosing the Replication Method can cause data discrepancies and latency.

This is a method in which modifications done to the source records like, updates, inserts and deletes are identified by reading Database’s binary log files. This method automatically detects the changes happen to the table structures in the source platform. When certain records are modified, the complete row is written to the log file as a log message. Dextrus takes the row-based approach and reads the log messages in sequence, meaning in the same order that the log messages were written to the log. Once this pipeline is scheduled to run, it keeps monitoring the db logs and keeps replicating the data to the target databases. So, it is pretty much real-time data replication process.

This is a method in which changed data is captured using the created on and changed on time stamp columns in the source tables. A bookmark timestamp is maintained to store the last extracted timestamp and by configuring the variables on the CDC columns, the next data set that is to be pulled can be identified by comparing the bookmark timestamp value. Once the pipeline is built, it can be scheduled to run it at frequency starting from once every minute. Depending on the expected data volumes for each pull, this latency can be configured properly.

This is a method in which the new and updated records are identified by using a column called a Replication Key. This Replication Key is usually a timestamp, date-time, or an integer column in the source table. Dextrus stores the max value of the table’s Replication Key Column for every fetch that happens on the source table. When the next fetch of data happens, it compares the max value of the Replication Key from the previous fetch and all the records that have key value greater or equal to the stored value are replicated and Dextrus stores the new maximum value and this process keeps repeating as long as the replication job keeps running.

Dextrus comes with several built-in connectors readily available to establish connectivity with ERP systems like SAP ECC, S4HANA, Salesforce etc…and Relational Databases, JDBC connectivity enabled systems, Modern Cloud data platforms, REST APIs, Social Media Platforms like Twitter and many more. Debezium, Confluent Kafka, Delta Lake are few of the important components that are integral part of robust and resilient architecture of Dextrus. May it be DB log based replication, CDC Query based replication or Key Based Incremental Replication, Dextrus has the suitable solutions to accommodate, based on the client’s choice of replication methods.