You can decide how the data will come to the catalog through the sync types. Whether you want only the updated data to be retrieved or the entire current status of the source, there is an option for you.

​
The types of sync

  1. Incremental: each time the extraction happens, we’ll fetch only the new data. This is ideal if, for example, your data flow is intense because the incremental sync makes the search for new data faster.
  • With this option, deleted data on your source will not be deleted on the lake until a full sync happens. Read about in on the Additional Full Sync section.
  1. Full sync: each time the extraction happens, we’ll fetch the current state of the data. This is ideal if, for example, you can not have in your catalog data that was aslready deleted on the source.

​
Additional full sync

If you choose an incremental sync for a set of data, you have the option to add an additional full sync. This allows you to periodically perform a complete extraction of the source state to the lake. We recommend that you choose the frequency of this operation wisely, as with any full sync. While it’s beneficial to always have all your data in sync with the source, it might not always be necessary. A full sync uses more time and resources compared to an incremental one. If you have a small number of records, this might not be an issue, but with many records, it can become unnecessarily costly. Consider your context and data volume before deciding.

TriggerRoles & Permissions