​
What are Delta Tables?

Delta Tables are a modern data storage format designed to bring reliability, scalability, and performance to your data workflows. They combine the best features of traditional tables with powerful new capabilities like version control, schema enforcement, and data consistency, making them ideal for analytics, machine learning, and more.

In Nekt, Delta Tables are automatically applied to data in the Raw Layer, providing a structured foundation for efficient querying and data lineage.

​
Why Use Delta Tables?

Delta Tables offer several advantages over traditional data storage formats:

  • Version Control: Track changes and access historical versions of your data with built-in versioning.
  • Schema Enforcement: Prevent accidental data corruption by ensuring the data adheres to a predefined structure.
  • Data Lineage: Maintain a clear audit trail of all data changes, essential for governance and compliance.
  • Efficient Querying: Optimize performance for large-scale analytics and machine learning workloads.
  • ACID Transactions: Ensure reliable and consistent data operations, even in complex workflows.

​
Key Features of Delta Tables

​
Version Control

Every update to a Delta Table creates a new version, enabling you to track changes over time. This makes Delta Tables ideal for scenarios like debugging, auditing, or reproducing past analyses.

​
Schema Evolution

Delta Tables can automatically detect and handle schema changes, allowing for seamless data updates while preserving structure and integrity.

​
Efficient Storage

By leveraging data compression and caching, Delta Tables reduce storage costs and improve read/write performance.

​
Time Travel

Delta Tables in Nekt support time travel, allowing you to explore and use previous versions of your tables with ease.

  • Delta Log Retention Duration: This defines the number of days previous versions of your table are accessible. By default, this is set to 30 days.
  • Delta Deleted File Retention Duration: This defines the number of days deleted files are retained after the Delta Log period ends. By default, this is set to 7 days.

​
Example:

Imagine your table undergoes updates daily:

  • For the first 30 days, you can navigate previous versions using time travel to inspect or restore older data.
  • After 30 days, files are retained for an additional 7 days, but accessing these files requires a request to us.

These settings can be customized for each table via the Settings tab, giving you control over how long data remains accessible.

​
How Delta Tables Work in Nekt

In Nekt, Delta Tables are automatically applied in the Raw Layer, ensuring your data starts with a reliable, scalable format. From there:

  1. Data is stored in Delta format: This ensures every action—insert, update, delete—is logged for consistency and tracking.
  2. Users manage transformations in other layers: Leveraging the structure provided by Delta Tables for clean and consistent workflows.
  3. Time Travel empowers recovery and analysis: Explore historical data versions as needed.

​
Configuring Delta Tables

In the Settings tab of each table, you can customize:

  • Delta Log Retention Duration: The number of days table versions are kept for navigation.
  • Delta Deleted File Retention Duration: The number of days deleted files are retained after the Delta Log period.

​
Conclusion

Delta Tables are the backbone of structured data management in Nekt. They provide a powerful combination of flexibility, performance, and reliability, ensuring your data

LayerTrigger