Announcing Ryft Data Retention & Compliance Enforcement for Apache Iceberg

Today, we’re introducing two new capabilities in Ryft: Automated Data Retention and Data Compliance Enforcement for Apache Iceberg™.

These features integrate directly into the Ryft platform to ensure efficient, policy-driven data deletion and compliance, working seamlessly alongside table maintenance and optimization.

Deleting Data in Iceberg Is Much Harder Than It Looks

Deleting data in Apache Iceberg is deceptively complex. Teams usually need to delete data for two reasons:

1. Cost & operational efficiency - drop old data to keep storage and compute costs predictable‍

2. Compliance - ensure regulated data is entirely removed (GDPR, CCPA, internal policies)

Both sound straightforward: “delete rows older than X” or “delete this user’s data.”

But with Iceberg, deletion needs to be done carefully and efficiently. Doing it wrong can corrupt tables, leave non-compliant data behind, or waste massive amounts of compute.

Customers required to comply with data regulations had to build custom workflows to locate soft deletes, consolidate them, rewrite affected partitions, and verify that old snapshots no longer reference the data.

Data Retention defines how long data should live: 7 days, 30 days, 13 months, etc. Although object storage systems support deletion policies, you can’t just use them directly on Iceberg tables, as the tables will get corrupted, because Iceberg metadata still references the files.

Data Compliance requirements like GDPR and CCPA demand physical deletion, not soft deletion. With data stored across thousands of files and referenced from older snapshots and backups, making sure Iceberg tables are compliant is an operational challenge.

How The Ryft Platform Solves It

1. Automated Data Retention

Ryft turns Iceberg retention into a fully automated, scalable operation. Instead of building custom pipelines or manually coordinating rewrites, you define the policy, and Ryft handles everything else.

Retention policies can be applied at any scope: a single table, a group of tables, or your entire lake.Ryft continuously evaluates tables and automatically identifies which partitions contain data past the retention window.

Data retention is performed in a partition-aware way, that is highly efficient as they do not require excessive data scans.

Automated Data Retention is also safe by default - dry runs and policy validation are performed to prevent accidental misconfiguration that can lead to data loss. Iceberg metadata is used to verify the expected amount of data to be deleted relative to the actual data in the table and makes sure there is no anomaly.

2. Compliance Enforcement

Ryft compliance cleanup consolidates all the complexity of keeping Iceberg tables compliant into a single checkbox.

You mark a table as “compliance-enforced,” and Ryft automatically ensures that any deletion is physically removed from the table: no special pipelines, no coordination with ingestion, no manual rewrites.

This works consistently even across thousands of tables, with different schemas, workloads, and ingestion patterns, both for live and historical data.

3. Unified With the Rest of Ryft

Retention and compliance enforcement aren't standalone features - they're integrated into Ryft's intelligent optimization engine.

Everything runs in coordination:

• Adaptive Optimization
• Retention policies
• Compliance cleanup
• Snapshot expiration
• Metadata cleanup

This means no conflicts or duplicate work, and no manual scheduling to prevent operations from stepping on each other.

From Operational Burden to Autonomous Infrastructure

Before Ryft, teams managing data deletion faced a choice:

• Build complex systems (months of engineering, ongoing maintenance)
• Accept non-compliance (regulatory risk, audit failures)

Ryft eliminates that choice. Automated retention and compliance enforcement handle what used to require dedicated engineering resources - and they scale to thousands of tables without additional overhead.

Our customers have already been running these in production with multi-petabyte lakes. Book a call with us to see it in action. These capabilities are available to all Ryft customers today. Whether you're managing compliance requirements or optimizing costs, we'll show you how to implement policies that deliver immediate results.