Problems while extracting Data from Source Systems for ETL Data Processing

The data extraction from the source system is very critical. Source Systems like EBS or any other source systems will be the production environment for Company. They facilitate their customers with the transactions everyday. So it’s important that we don’t affect the business.

When we extract data from the source and use it for Transformation and loading in our data staging area, at any cost it should not affect the Source System .So it’s most important to avoid overloading of the source while extracting.

Most of the source systems will be legacy systems and hence we cannot interfere much into changing the existing configuration of the system for better performance while extracting.

There are many technical solutions to avoid this situation. One way is to use Incremental loading, Extract only those records with timestamps greater than the one of the last extraction. Or else we need to implement Triggers to inform whenever source modification happens, i.e. any updates or inserts or another activity takes place.