## Ryft Adaptive Optimization [Ryft Blog](https://www.ryft.io/blog) News # Announcing Ryft Adaptive Optimization Today, we’re officially introducing **Ryft Adaptive Optimization** \- always-on, dynamic optimization engine for Apache Iceberg™. Our engine continuously compacts, rewrites, indexes, and reorders data based on how your tables are actually used, delivering up to **5× faster queries**, **10x storage reduction**, and **7x better compaction efficiency** compared to other engines. ## **The Challenge with Manual Table Management** Managing Iceberg tables is an extremely important yet manual process requiring a lot of testing and constant tweaking to get right. Data teams need to constantly ensure that each table is maintained and optimized, otherwise performance would plummet and costs would skyrocket. The challenge is that every table is unique: 1. **Different workload patterns:** Batch processing, CDC streams, real-time ingestion, bulk merges, and large deletes all have distinct optimization needs. 2. **Varying data characteristics**: Wide tables, highly compressible datasets, JSON columns, and high or low cardinality fields each require different approaches. 3. **Unpredictable access patterns**: Query filters, group-bys, and joins constantly evolve, making static optimization strategies obsolete. This forces teams into endless cycles of monitoring and manual tuning - adjusting compaction schedules, debugging failed maintenance jobs, tweaking compression settings, optimizing file sizes, indexing columns, and reordering data. It's time-consuming, error-prone, and doesn't scale - not a good spend of your time. ## **How Adaptive Optimization Works** Ryft's engine learns the unique characteristics of each table and automatically adapts optimization strategies accordingly. We analyze three key dimensions: 1. **Workload type:** Streaming workloads need different compaction algorithms and schedules than batch processing. 2. **Data profiling**: We continuously assess data volumes, compression ratios, column counts, and distribution patterns. 3. **Usage analytics:** By collecting query patterns from your compute engines, we make informed decisions about how to optimize each table for maximum impact. **Looking Under The Covers** - **Targeted partition compaction:** Only compact partitions that need it, based on file size, age, delete density, and scan frequency. - **Predictive efficiency scoring:** Evaluate optimization plans upfront and skip work that won't meaningfully improve performance or cost. - **Intelligent delete file management:** Rewrite equality and position deletes at the right time and for the right partitions, and clean up orphaned delete files. - **Streaming-aware tiered compaction:** Micro-merges for frequently accessed data, with opportunistic macro-compaction that preserves SLAs. - **Dynamic resource allocation:** Dynamically allocates resources based on data volume and properties, to achieve the best performance and cost. - **Smart partition prioritization:** Partitions are prioritized for optimization based on recency, usage, and potential benefit. - **Metadata optimization:** Reduce query planning overhead by compacting manifests and managing metadata overhead. - **Dynamic file sizing:** Automatically tune target file sizes based on usage patterns - wider files for scan-heavy fact tables, smaller files for point lookups. ## **Real-World Impact** ### Optimize Only What Matters For organizations managing hundreds, thousands or tens of thousands of tables, not every table needs constant optimization. Tables that update infrequently don't need constant compaction. Tables created and not updated or rarely used don't need ongoing maintenance. Ryft’s engine identifies unused tables, optimizes them first and skips unnecessary, ongoing optimizations, ensuring effective use of precious compute resources and budgets. ### CDC Replication Made Simple Change Data Capture workloads present two unique challenges. - Many different partitions are frequently updated, making naive "last partition" optimization strategies ineffective. - Engineers need to monitor and fix commit conflicts when compaction jobs interfere with ongoing table updates. Ryft monitors all partitions individually, triggering optimization only where changes occurred, while running specialized jobs to minimize commit conflicts and maintain table health. ### Streaming Without the Headaches Streaming workloads create two persistent problems: endless small files from frequent micro-batches, and commit conflicts between compaction and active writers. Ryft’s engine detects streaming patterns and adapts its behavior to maintain read performance without interfering with data ingestion, ensuring your streaming pipelines stay healthy and performant. ### **Seamlessly Integrated with Compliance Rules** For tables requiring GDPR or CCPA compliance, and ongoing retention and tiering, Ryft’s engine coordinates optimization with data retention and compliance cleanup policies. This prevents job collisions and ensures all maintenance tasks are performed efficiently within a unified system. ### **Proven at Scale** Ryft Adaptive Optimization has been battle-tested in production with several large, multi-petabyte data lakes. The results consistently demonstrate: - **Up to 5× faster queries** on frequently accessed partitions through targeted compaction and sort rewrites. - **~7× higher compaction efficiency** compared to other engines, delivering more performance per byte rewritten. - **Up to 10× storage reduction** through adaptive compression and comprehensive cleanup of snapshots, small files, metadata, and delete files. Bytes processed by compaction for a 500GB streaming table ingested over a day. ![](https://cdn.prod.website-files.com/68ef977c5ec5483ef208e8b3/68efaaa3554eca8245e5f2d0_68caa477218cfe7f1942b8b2_Effeciency%2520(1).avif) Bytes processed by compaction for a 500GB streaming table ingested over a day. ![](https://cdn.prod.website-files.com/68ef977c5ec5483ef208e8b3/68efaaa3554eca8245e5f2cb_68caa4df43c652145d5153a5_Query%2520Performance.avif) Bytes scanned by queries from a table, before and after enabling Ryft Adaptive Optimization. ## **Ryft Adaptive Optimization is now available for everyone** Adaptive Optimization is generally available in **Ryft** for all of our customers. Unlock faster queries, predictable SLAs, and lower compute and storage costs, without the manual overhead. [**Contact us**](https://ryft.io/demo) to enable Adaptive Optimization on your highest-value tables and see the impact in your environment. Table of Contents [The Challenge with Manual Table Management](https://www.ryft.io/blog/announcing-ryft-adaptive-optimization#the-challenge-with-manual-table-management) [How Adaptive Optimization Works](https://www.ryft.io/blog/announcing-ryft-adaptive-optimization#how-adaptive-optimization-works) [Real-World Impact](https://www.ryft.io/blog/announcing-ryft-adaptive-optimization#real-world-impact) [Optimize Only What Matters](https://www.ryft.io/blog/announcing-ryft-adaptive-optimization#optimize-only-what-matters) [CDC Replication Made Simple](https://www.ryft.io/blog/announcing-ryft-adaptive-optimization#cdc-replication-made-simple) [Streaming Without the Headaches](https://www.ryft.io/blog/announcing-ryft-adaptive-optimization#streaming-without-the-headaches) [Seamlessly Integrated with Compliance Rules](https://www.ryft.io/blog/announcing-ryft-adaptive-optimization#seamlessly-integrated-with-compliance-rules) [Proven at Scale](https://www.ryft.io/blog/announcing-ryft-adaptive-optimization#proven-at-scale) [Ryft Adaptive Optimization is now available for everyone](https://www.ryft.io/blog/announcing-ryft-adaptive-optimization#ryft-adaptive-optimization-is-now-available-for-everyone) Get the latest posts straight to your inbox Thank you! Your submission has been received! ## Browse other blogs ![Announcing Ryft Data Retention & Compliance Enforcement for Apache Iceberg](https://cdn.prod.website-files.com/68ef977c5ec5483ef208e8b3/6936da25feec7bdfd478e008_Blog%20Cover.png) News ## Announcing Ryft Data Retention & Compliance Enforcement for Apache Iceberg Today, we’re introducing two new capabilities in Ryft: Automated Data Retention and Data Compliance Enforcement for Apache Iceberg™. These features integrate directly into the Ryft platform to ensure efficient, policy-driven data deletion and compliance, working seamlessly alongside table maintenance and optimization. ![](https://cdn.prod.website-files.com/68ef977c5ec5483ef208e8b3/68f105fe51b1bd3a206f8f26_Auteur_pic_01%20(2).avif) Yuval Yogev December 8, 2025 December 8, 2025 [blog](https://www.ryft.io/blog/announcing-ryft-data-retention-compliance-enforcement-for-apache-iceberg) ![Announcing Ryft Adaptive Optimization](https://cdn.prod.website-files.com/68ef977c5ec5483ef208e8b3/68efaaa4554eca8245e5f2f2_68caa4aa095abc61c9362c0b_Blog%2520Cover%2520(1).avif) News ## Announcing Ryft Adaptive Optimization Today, we’re officially introducing Ryft Adaptive Optimization - always-on, dynamic optimization engine for Apache Iceberg™. Our engine continuously compacts, rewrites, indexes, and reorders data based on how your tables are actually used, delivering up to 5× faster queries, 10x storage reduction, and 7x better compaction efficiency compared to other engines. ![](https://cdn.prod.website-files.com/68ef977c5ec5483ef208e8b3/68f105da696b0f9017b6c9ed_Auteur_pic_01.avif) Yossi Reitblat October 23, 2025 September 17, 2025 [blog](https://www.ryft.io/blog/announcing-ryft-adaptive-optimization) ![Unlocking Iceberg management for everyone](https://cdn.prod.website-files.com/68ef977c5ec5483ef208e8b3/68efac34c4925bbf47b6a113_689363b8bb6dd97ccd38b882_686d09873da2afe9164ed0ba_cover.avif) News ## Unlocking Iceberg management for everyone Ryft, in many ways, is a story 15 years in the making. Yuval Yogev, Guy Gadon and I went to the same high school, worked together at 8200, building high-scale data infrastructure, and went our separate ways - all to realize that we really enjoy solving complicated data infrastructure problems together with the people we love. ![](https://cdn.prod.website-files.com/68ef977c5ec5483ef208e8b3/68f105da696b0f9017b6c9ed_Auteur_pic_01.avif) Yossi Reitblat October 27, 2025 July 8, 2025 [blog](https://www.ryft.io/blog/unlocking-iceberg-management-for-everyone) ## See Ryft in Action The only solution that automatically maintains and optimizes your Iceberg tables based on usage, no lock-in, no manual tuning [See Ryft in Action](https://www.ryft.io/see-ryft-in-action) ![screening illustration](https://cdn.prod.website-files.com/68cf7feaa53a47caf3256f08/68ee7e458fe1bc5e963b6380_Frame%201707484444.avif) ![blog cta ](https://cdn.prod.website-files.com/68cf7feaa53a47caf3256f08/68efc958cdd00391780e03c2_Frame%201707484444%20(1).avif) [home](https://www.ryft.io/) ## Ryft Blog Insights [blog](https://www.ryft.io/blog/announcing-ryft-adaptive-optimization) ![](https://cdn.prod.website-files.com/68ef977c5ec5483ef208e8b3/68efaaa4554eca8245e5f2f2_68caa4aa095abc61c9362c0b_Blog%2520Cover%2520(1).avif) News featured ## Announcing Ryft Adaptive Optimization Today, we’re officially introducing Ryft Adaptive Optimization - always-on, dynamic optimization engine for Apache Iceberg™. Our engine continuously compacts, rewrites, indexes, and reorders data based on how your tables are actually used, delivering up to 5× faster queries, 10x storage reduction, and 7x better compaction efficiency compared to other engines. ![Yossi Reitblat](https://cdn.prod.website-files.com/68ef977c5ec5483ef208e8b3/68f105da696b0f9017b6c9ed_Auteur_pic_01.avif) Yossi Reitblat October 23, 2025 September 17, 2025 ![Apache Iceberg V3: Is It Ready?](https://cdn.prod.website-files.com/68ef977c5ec5483ef208e8b3/69415e4b45acad26189a17d2_Blog%20Coverv3.png) Engineering ## Apache Iceberg V3: Is It Ready? Apache Iceberg V3 is a huge step forward for the lakehouse ecosystem. The V3 specification was finalized and ratified earlier this year, bringing several long-awaited capabilities into the core of the format: efficient row-level deletes, built-in row lineage, better handling of semi-structured data, and the beginnings of native encryption. This post breaks down the major features, the current state of implementation, and what this means for real adoption. ![](https://cdn.prod.website-files.com/68ef977c5ec5483ef208e8b3/68f1061b22ca6c79303ddfab_Auteur_pic_01%20(3).avif) Guy Yasoor December 16, 2025 December 17, 2025 [blog](https://www.ryft.io/blog/apache-iceberg-v3-is-it-ready) ![Announcing Ryft Data Retention & Compliance Enforcement for Apache Iceberg](https://cdn.prod.website-files.com/68ef977c5ec5483ef208e8b3/6936da25feec7bdfd478e008_Blog%20Cover.png) News ## Announcing Ryft Data Retention & Compliance Enforcement for Apache Iceberg Today, we’re introducing two new capabilities in Ryft: Automated Data Retention and Data Compliance Enforcement for Apache Iceberg™. These features integrate directly into the Ryft platform to ensure efficient, policy-driven data deletion and compliance, working seamlessly alongside table maintenance and optimization. ![](https://cdn.prod.website-files.com/68ef977c5ec5483ef208e8b3/68f105fe51b1bd3a206f8f26_Auteur_pic_01%20(2).avif) Yuval Yogev December 8, 2025 December 8, 2025 [blog](https://www.ryft.io/blog/announcing-ryft-data-retention-compliance-enforcement-for-apache-iceberg) ![Streaming with Apache Iceberg: The Operational Problems at Scale](https://cdn.prod.website-files.com/68ef977c5ec5483ef208e8b3/692895683e8be6007b3e1902_Blog%20Cover.png) Engineering ## Streaming with Apache Iceberg: The Operational Problems at Scale Streaming into Iceberg creates three operational problems most teams don't see coming: small files pile up faster than you can compact them, storage costs climb because you're paying for data you've already replaced, and merges take much longer than they should. After seeing these problems in real production data lakes, we decided to share more about the causes and possible solutions. ![](https://cdn.prod.website-files.com/68ef977c5ec5483ef208e8b3/68f105fe51b1bd3a206f8f26_Auteur_pic_01%20(2).avif) Yuval Yogev December 8, 2025 December 1, 2025 [blog](https://www.ryft.io/blog/streaming-with-apache-iceberg-the-operational-problems-at-scale) ![How to Choose an Apache Iceberg Catalog](https://cdn.prod.website-files.com/68ef977c5ec5483ef208e8b3/692068eb2d13507cc7245b1e_Blog%20Cover.png) Engineering ## How to Choose an Apache Iceberg Catalog Apache Iceberg has become the table format of choice for building open data lakehouses. It solves long-standing problems around ACID transactions and engine interoperability. This post covers how to choose an Iceberg Catalog for production. ![](https://cdn.prod.website-files.com/68ef977c5ec5483ef208e8b3/68f1061b22ca6c79303ddfab_Auteur_pic_01%20(3).avif) Guy Yasoor December 8, 2025 November 21, 2025 [blog](https://www.ryft.io/blog/how-to-choose-an-apache-iceberg-catalog) ![How to Fix Corrupted Iceberg Tables](https://cdn.prod.website-files.com/68ef977c5ec5483ef208e8b3/690a18544f72106efe53dba3_New%20blog%20cover.png) Engineering ## How to Fix Corrupted Iceberg Tables In Part 1 and Part 2 of this series, we analyzed two different scenarios that led to Iceberg table corruption - from silent overwrites to inconsistent metadata. Since publishing these posts we have received more requests from people who encountered these situations on how to safely repair those tables. In this post, we’ll focus on the remediation process: identifying what’s affected, how to safely clean it up, and how to prevent further damage. ![](https://cdn.prod.website-files.com/68ef977c5ec5483ef208e8b3/68f105eb118c0310c1754068_Auteur_pic_01%20(1).avif) Omer Hadari November 7, 2025 November 4, 2025 [blog](https://www.ryft.io/blog/how-to-fix-corrupted-iceberg-tables) ![Handling Commit Conflicts in Apache Iceberg: Patterns and Fixes](https://cdn.prod.website-files.com/68ef977c5ec5483ef208e8b3/68fa1827f28db910371a26d5_commit_conflict_cover.png) Engineering ## Handling Commit Conflicts in Apache Iceberg: Patterns and Fixes Commit conflicts in Apache Iceberg are one of those problems that seem rare - until you start operating at scale. The first time a long-running compaction job fails after hours of compute, or a CDC pipeline spends half its time retrying commits, you realize this isn’t a corner case. It’s a core operational challenge that directly impacts cost, latency, and reliability.This post covers what commit conflicts are, why they happen, and how to fix them without creating new problems in the process. ![](https://cdn.prod.website-files.com/68ef977c5ec5483ef208e8b3/68f105da696b0f9017b6c9ed_Auteur_pic_01.avif) Yossi Reitblat October 23, 2025 October 23, 2025 [blog](https://www.ryft.io/blog/handling-commit-conflicts-in-apache-iceberg-patterns-and-fixes) ![Data Retention in Apache Iceberg: Implementation Details and Best Practices](https://cdn.prod.website-files.com/68ef977c5ec5483ef208e8b3/68efaaa43a9a4f90a3cc617a_68e01bb701e38e364ba20180_retention_cover.avif) Engineering ## Data Retention in Apache Iceberg: Implementation Details and Best Practices Data retention in Apache Iceberg is one of those critical operations that seems simple until you implement it at scale. Delete old data, save money, stay compliant - straightforward enough. But the implementation details matter, and getting them wrong can mean failed compliance audits, runaway storage costs, or accidentally purging the wrong data. ![](https://cdn.prod.website-files.com/68ef977c5ec5483ef208e8b3/68f105fe51b1bd3a206f8f26_Auteur_pic_01%20(2).avif) Yuval Yogev October 23, 2025 October 3, 2025 [blog](https://www.ryft.io/blog/data-retention-in-apache-iceberg-implementation-details-and-best-practices) ![Announcing Ryft Adaptive Optimization](https://cdn.prod.website-files.com/68ef977c5ec5483ef208e8b3/68efaaa4554eca8245e5f2f2_68caa4aa095abc61c9362c0b_Blog%2520Cover%2520(1).avif) News ## Announcing Ryft Adaptive Optimization Today, we’re officially introducing Ryft Adaptive Optimization - always-on, dynamic optimization engine for Apache Iceberg™. Our engine continuously compacts, rewrites, indexes, and reorders data based on how your tables are actually used, delivering up to 5× faster queries, 10x storage reduction, and 7x better compaction efficiency compared to other engines. ![](https://cdn.prod.website-files.com/68ef977c5ec5483ef208e8b3/68f105da696b0f9017b6c9ed_Auteur_pic_01.avif) Yossi Reitblat October 23, 2025 September 17, 2025 [blog](https://www.ryft.io/blog/announcing-ryft-adaptive-optimization) ![Iceberg Table Corruption and Data Loss in the Wild: Part 2](https://cdn.prod.website-files.com/68ef977c5ec5483ef208e8b3/68efac2f73600f936c593ae1_68a5ba8ec3df6d4e0e4d9519_blog.avif) Engineering ## Iceberg Table Corruption and Data Loss in the Wild: Part 2 Data and metadata integrity issues in Iceberg, particularly in streaming workloads, often present similar patterns. In these two posts we covered two similar Iceberg table corruption issues, which manifested in almost the exact same way - data file overwrites. Each time, the underlying reason was different. ![](https://cdn.prod.website-files.com/68ef977c5ec5483ef208e8b3/68f105eb118c0310c1754068_Auteur_pic_01%20(1).avif) Omer Hadari October 21, 2025 August 20, 2025 [blog](https://www.ryft.io/blog/iceberg-table-corruption-and-data-loss-in-the-wild-part-2) ![Why Apache Iceberg Finally Unlocks Security Data Lakes](https://cdn.prod.website-files.com/68ef977c5ec5483ef208e8b3/68efaaad2cfdee77c8cfca4d_68934f1203ec38af84c342dc_cyber_blog_cover.avif) Engineering ## Why Apache Iceberg Finally Unlocks Security Data Lakes Security data lakes are notoriously hard to build and operate efficiently. Apache Iceberg changes the equation. It’s an open table format purpose-built for scalable, flexible, and cost-effective analytics - and it’s quickly becoming the new standard for modern cybersecurity data lakes.Here’s why. ![](https://cdn.prod.website-files.com/68ef977c5ec5483ef208e8b3/68f105fe51b1bd3a206f8f26_Auteur_pic_01%20(2).avif) ![](https://cdn.prod.website-files.com/68ef977c5ec5483ef208e8b3/68f1061b22ca6c79303ddfab_Auteur_pic_01%20(3).avif) Yuval Yogev Guy Yasoor October 23, 2025 August 6, 2025 [blog](https://www.ryft.io/blog/why-apache-iceberg-finally-unlocks-security-data-lakes) ![GDPR Compliance with Apache Iceberg: A Practical Guide](https://cdn.prod.website-files.com/68ef977c5ec5483ef208e8b3/68efac2e7d5065b5e6763571_688a5738f58dabd4b258cd0b_Blog%2520Cover.avif) Engineering ## GDPR Compliance with Apache Iceberg: A Practical Guide GDPR compliance boils down to one critical requirement: when a user requests deletion of their data, you must delete ALL traces of their “user identifiable information” across ALL systems and copies. Not hide it. Not mark it as deleted. Delete it completely ![](https://cdn.prod.website-files.com/68ef977c5ec5483ef208e8b3/68f105da696b0f9017b6c9ed_Auteur_pic_01.avif) Yossi Reitblat October 23, 2025 July 30, 2025 [blog](https://www.ryft.io/blog/gdpr-compliance-with-apache-iceberg-a-practical-guide) ![High performant graph queries on Apache Iceberg powered by Ryft and PuppyGraph](https://cdn.prod.website-files.com/68ef977c5ec5483ef208e8b3/68efac2d996c172ac356255e_687e95e55a1d256fe9809cd0_puppy_cover.avif) News ## High performant graph queries on Apache Iceberg powered by Ryft and PuppyGraph Graph workloads traditionally rely on specialized graph storage systems, but these come with significant challenges in scalability, performance, and data duplication. By combining the storage optimization capabilities of Ryft on Apache Iceberg with PuppyGraph’s advanced graph query engine, teams can run high-performance, scalable graph queries directly on their Iceberg data lake - without moving or duplicating data. ![](https://cdn.prod.website-files.com/68ef977c5ec5483ef208e8b3/68f105da696b0f9017b6c9ed_Auteur_pic_01.avif) Yossi Reitblat October 23, 2025 July 21, 2025 [blog](https://www.ryft.io/blog/high-performant-graph-queries-on-apache-iceberg-powered-by-ryft-and-puppygraph) ![Iceberg Table Corruption and Data Loss in the Wild: Part 1](https://cdn.prod.website-files.com/68ef977c5ec5483ef208e8b3/68efac2d7195a84051d2f0fa_68779b8f2dbafc67dd6d41b4_New%2520blog%2520cover.avif) Engineering ## Iceberg Table Corruption and Data Loss in the Wild: Part 1 In this post, we want to share a story about a sneaky bug we encountered that caused table corruption, as well as silent data loss in Iceberg tables. If you're using Iceberg, if your ingestion is based on a streaming pipeline, if you're an AWS EMR user, or if you just like a good bug hunt - read on. ![](https://cdn.prod.website-files.com/68ef977c5ec5483ef208e8b3/68f105eb118c0310c1754068_Auteur_pic_01%20(1).avif) Omer Hadari October 27, 2025 July 16, 2025 [blog](https://www.ryft.io/blog/iceberg-table-corruption-and-data-loss-in-the-wild-part-1) ![Ryft Raises $8M to Help Enterprises Take Control Over Their Data](https://cdn.prod.website-files.com/68ef977c5ec5483ef208e8b3/68efac2f1f7a3dc4aff4067d_686e9e350fcc90a07fb26c66_Ryft%2520Group%2520Photo%2520(1)%25201.avif) News ## Ryft Raises $8M to Help Enterprises Take Control Over Their Data For years, cloud giants like Snowflake, Databricks, Microsoft, and Google have made billions by offering enterprises an easy way to store and analyze data, as long as that data stays within their platforms. But that convenience came at a hidden cost: soaring expenses, rigid infrastructure, and deep vendor lock-in that slowed innovation and made AI adoption harder. Still, many companies remain stuck in outdated systems, deterred by the complexity of managing their data independently. ![](https://cdn.prod.website-files.com/68ef977c5ec5483ef208e8b3/68f105da696b0f9017b6c9ed_Auteur_pic_01.avif) Yossi Reitblat October 23, 2025 July 9, 2025 [blog](https://www.ryft.io/blog/ryft-raises-8m-to-help-enterprises-take-control-over-their-data) ![Unlocking Iceberg management for everyone](https://cdn.prod.website-files.com/68ef977c5ec5483ef208e8b3/68efac34c4925bbf47b6a113_689363b8bb6dd97ccd38b882_686d09873da2afe9164ed0ba_cover.avif) News ## Unlocking Iceberg management for everyone Ryft, in many ways, is a story 15 years in the making. Yuval Yogev, Guy Gadon and I went to the same high school, worked together at 8200, building high-scale data infrastructure, and went our separate ways - all to realize that we really enjoy solving complicated data infrastructure problems together with the people we love. ![](https://cdn.prod.website-files.com/68ef977c5ec5483ef208e8b3/68f105da696b0f9017b6c9ed_Auteur_pic_01.avif) Yossi Reitblat October 27, 2025 July 8, 2025 [blog](https://www.ryft.io/blog/unlocking-iceberg-management-for-everyone) ![Athena vs. Snowflake on Iceberg: Performance and Cost Comparison on TPC-H](https://cdn.prod.website-files.com/68ef977c5ec5483ef208e8b3/68efaaa40baa4b79fdea5613_689363feed050d2145cbe300_68403b6d7aafbecebd5589e2_683b6564cd8fe3c5baf29a65_Option%25201.avif) Engineering ## Athena vs. Snowflake on Iceberg: Performance and Cost Comparison on TPC-H How do Amazon Athena and Snowflake compare when running real-world analytics on Apache Iceberg tables? We ran a TPC-H benchmark to break down the trade-offs in performance, cost, and architecture. ![](https://cdn.prod.website-files.com/68ef977c5ec5483ef208e8b3/68f105fe51b1bd3a206f8f26_Auteur_pic_01%20(2).avif) ![](https://cdn.prod.website-files.com/68ef977c5ec5483ef208e8b3/68f105da696b0f9017b6c9ed_Auteur_pic_01.avif) Yuval Yogev Yossi Reitblat October 27, 2025 May 26, 2025 [blog](https://www.ryft.io/blog/athena-vs-snowflake-on-iceberg-tpch-comparison) ![Making Sense of Apache Iceberg Statistics](https://cdn.prod.website-files.com/68ef977c5ec5483ef208e8b3/68efac2e5e21b981698a1e96_68403bf334ef9009d11fae53_683b6409f6c82555d8e9b635_cover%2520post%2520(1).avif) Engineering ## Making Sense of Apache Iceberg Statistics Apache Iceberg™ is known for its rich metadata model, and one of its most powerful (but often confusing) features is its support for statistics. In this blog post we will break them down, helping you understand what exists today, what you should configure, and what’s coming next. ![](https://cdn.prod.website-files.com/68ef977c5ec5483ef208e8b3/68f1061b22ca6c79303ddfab_Auteur_pic_01%20(3).avif) Guy Yasoor October 27, 2025 May 26, 2025 [blog](https://www.ryft.io/blog/making-sense-of-apache-iceberg-statistics) ### Oops! No matches found Try tweaking your filters or resetting them to see everything again. [Reset All](https://www.ryft.io/blog#) [home](https://www.ryft.io/) ## Choosing Iceberg Catalog [Ryft Blog](https://www.ryft.io/blog) Engineering # How to Choose an Apache Iceberg Catalog ![](https://cdn.prod.website-files.com/68ef977c5ec5483ef208e8b3/68f1061b22ca6c79303ddfab_Auteur_pic_01%20(3).avif) Guy Yasoor November 21, 2025 ![](https://cdn.prod.website-files.com/68ef977c5ec5483ef208e8b3/692068eb2d13507cc7245b1e_Blog%20Cover.png) Apache Iceberg has become the table format of choice for building open data lakehouses. It solves long-standing problems around ACID transactions and engine interoperability. The table format itself is standardized, but what is often overlooked when making architectural decisions is the catalog layer - the component that tracks and exposes table metadata. Your catalog can influence query performance, access control, and which engines can write to your tables. It also determines how much vendor lock-in risk you take on. This breakdown examines what catalogs do, how the REST specification has impacted them, and how to evaluate your options. ## How Iceberg Catalogs Work Every Iceberg deployment requires a catalog to manage transactions and track metadata. Iceberg tables store data in Parquet files, with metadata stored in separate metadata files. Each time a table is modified, Iceberg writes a new metadata.json file. Over time, you accumulate versions: v1.metadata.json, v2.metadata.json, and so on. The catalog is responsible for tracking which metadata.json file is the current one. ![](https://cdn.prod.website-files.com/68ef977c5ec5483ef208e8b3/692068ed36ab6198a9af3851_90095810.png) When [Spark](https://spark.apache.org/) or [Trino](https://trino.io/) needs to read a table, it asks the catalog for the current metadata pointer. When a writer commits new data, the catalog performs an atomic compare-and-swap: it reads the current pointer, verifies it hasn't changed, then updates to the new metadata file. This compare-and-swap handles concurrent writes. When two writers commit simultaneously, one succeeds and the other fails with a [commit conflict](https://www.ryft.io/blog/handling-commit-conflicts-in-apache-iceberg-patterns-and-fixes). The catalog is also responsible to determine whether a specific client can access a certain table, and manage the access control of the data. ## The REST Catalog Specification Before the REST catalog spec, each catalog needed engine-specific support. [Hive Metastore](https://hive.apache.org/) required a Java client, while [AWS Glue](https://docs.aws.amazon.com/glue/latest/dg/what-is-glue.html) needed SDK integration. Custom catalogs required adding external modules to every engine's runtime. The Iceberg REST Catalog specification (introduced in Iceberg 0.14.0) standardized how engines interact with catalogs through HTTP-based APIs. Any engine supporting the REST spec can work with any REST-compatible catalog. All major Iceberg implementations - Java, Python, Rust, and Go - support the REST Catalog specification. Engines can read and write tables without embedding the catalog, and catalog functions like permissions, optimizations, and validations run server-side instead of being duplicated in each engine. The REST spec has become the standard across products that work with Iceberg tables. Dremio, Snowflake, Google BigLake, AWS Glue, and Unity Catalog all support the REST Catalog spec, so engine choice can follow workload requirements rather than platform restrictions. ## How to Evaluate Iceberg Catalogs Catalogs differ in how they implement and support different capabilities. These are the key points to consider when comparing them: ### 1\. RBAC (Role-Based Access Control) Access control determines who can read or write data, create or drop tables, and manage table configuration. Some catalogs, like [Unity Catalog](https://www.databricks.com/product/unity-catalog) and [Apache Polaris](https://polaris.apache.org/) provide native table level access. Others like AWS Glue and Hive Metastore rely on external systems like [AWS IAM](https://aws.amazon.com/iam/) or [Apache Ranger](https://ranger.apache.org/). Integration with existing identity providers matters. This includes LDAP, OAuth, or SAML. If your catalog can't connect to your current auth system, you'll end up managing separate identities. ### 2\. Access Auditing Beyond controlling who can access tables, you need to track how the tables are being used. Audit logs are a compliance requirement for most production systems. AWS Glue, for example, integrates with [AWS CloudTrail](https://docs.aws.amazon.com/awscloudtrail/latest/userguide/cloudtrail-user-guide.html) to track requests to the catalog. Hive Metastore and Apache Polaris require external monitoring or custom instrumentation to track activity. ### 3\. Deployment & Scale Most catalogs handle 5-10 concurrent clients per table without issues, but some start showing contention or slower commits as concurrency grows. Glue depends on AWS API limits that vary by account and region. Hive Metastore runs into database contention as commits lock rows. Stateless catalogs like Polaris or [Nessie](https://projectnessie.org/) scale horizontally by adding instances behind a load balancer, or by provisioning more resources. Serverless or managed catalogs offer easier operation and scaling. If on-premise or custom tuning is necessary, self-hosting an open-source catalog is an option, though it demands maintenance and resource provisioning for scaling. ### 4\. Neutrality & REST Support One of the main benefits of Iceberg is its engine interoperability. However, some catalogs limit the engines that can write or read from them. Choosing a fully open catalog allows your lakehouse architecture to be fully flexible and future-proof. Full REST catalog specification support means any engine can read and write tables without proprietary SDKs. Some catalogs implement the complete spec while others have partial support or require specific engine integrations. ### 5\. Product Maturity When selecting an Iceberg catalog, prioritize solutions that are mature, production-ready, and widely adopted, as the catalog space is rapidly evolving. A popular choice is more likely to incorporate new Iceberg features quickly. If you opt for an open-source tool, assess the activity and size of its community to make sure bugs are resolved and new features are adopted quickly. ## Iceberg Catalog Options Your catalog choice depends on the infrastructure and workload requirements. At a high level, catalogs fall into a few categories. ### Legacy Catalogs Legacy lakehouses often use catalogs that originally supported older lake architectures, and adapted Apache Iceberg later on. The most prominent example is [Hive Metastore](https://hive.apache.org/), which was the original catalog for Apache Hive, and later adapted for Iceberg. Use it only if you're migrating from Hive and when the switch would be more disruptive than keeping it in place. For new deployments, pick a modern catalog. ### Cloud Vendors All major cloud providers now offer an Iceberg catalog of their own. Choose these when your architecture and access are fully incorporated into one cloud provider. The clearest example is [AWS Glue](https://aws.amazon.com/glue/), a data catalog fully managed by AWS. Access is managed via IAM, with auxiliary services like AWS Lake Formation for advanced access control. Microsoft offers an Iceberg catalog named OneLake deeply integrated with the Azure and Fabric ecosystems, while Google offers BigLake Metastore for GCP lakehouses based on GCS. ### Open Source Catalogs As part of the Iceberg ecosystem, many catalogs are developed open-source. The most popular catalog in this category is [Apache Polaris](https://polaris.apache.org/) (incubating), which was originally open sourced by Snowflake. Polaris follows the REST catalog specification and has an active community that continuously adds advanced features like advanced RBAC and support for other open table formats. Other open source catalogs include: - Unity Catalog - an open-source version of Databricks’ proprietary catalog - Nessie - A Git-style version control catalog - Lakekeeper - offers advanced access control - Apache Gravitino - a geo-distributed data catalog with Iceberg support ## Feature Comparison ![](https://cdn.prod.website-files.com/68ef977c5ec5483ef208e8b3/692068ae510a2cb3ca8c707f_Table.png) Apache Iceberg Catalogs Feature Comparison ## Wrapping Up Iceberg catalogs are a core part of any modern lakehouse. Most catalogs today implement the REST spec, and fulfill the basic requirements of handling metadata and coordinating commits. However, differences in deployment, monitoring, and access control, are critical when choosing the right one. Focus on openness to avoid vendor or cloud lock-in, and on product maturity to ensure new capabilities land quickly and reliably. The responsibility of catalogs is expected to grow even more with new community proposals such as: - **FGAC (fine-grained access control)** \- defining row-level and column-level access policies (e.g., _“user X can only access rows where region = Y”_). - **Server-side planning** \- shifting scan planning into the catalog, so optimizations happen centrally instead of in each engine. The catalog is a central piece of your architecture and affects many aspects of your lakehouse. Understanding the differences will help you choose the option that best supports your architecture and long-term plans. Table of Contents [How Iceberg Catalogs Work](https://www.ryft.io/blog/how-to-choose-an-apache-iceberg-catalog#how-iceberg-catalogs-work) [The REST Catalog Specification](https://www.ryft.io/blog/how-to-choose-an-apache-iceberg-catalog#the-rest-catalog-specification) [How to Evaluate Iceberg Catalogs](https://www.ryft.io/blog/how-to-choose-an-apache-iceberg-catalog#how-to-evaluate-iceberg-catalogs) [1\. RBAC (Role-Based Access Control)](https://www.ryft.io/blog/how-to-choose-an-apache-iceberg-catalog#1-rbac-role-based-access-control) [2\. Access Auditing](https://www.ryft.io/blog/how-to-choose-an-apache-iceberg-catalog#2-access-auditing) [3\. Deployment & Scale](https://www.ryft.io/blog/how-to-choose-an-apache-iceberg-catalog#3-deployment-and-scale) [4\. Neutrality & REST Support](https://www.ryft.io/blog/how-to-choose-an-apache-iceberg-catalog#4-neutrality-and-rest-support) [5\. Product Maturity](https://www.ryft.io/blog/how-to-choose-an-apache-iceberg-catalog#5-product-maturity) [Iceberg Catalog Options](https://www.ryft.io/blog/how-to-choose-an-apache-iceberg-catalog#iceberg-catalog-options) [Legacy Catalogs](https://www.ryft.io/blog/how-to-choose-an-apache-iceberg-catalog#legacy-catalogs) [Cloud Vendors](https://www.ryft.io/blog/how-to-choose-an-apache-iceberg-catalog#cloud-vendors) [Open Source Catalogs](https://www.ryft.io/blog/how-to-choose-an-apache-iceberg-catalog#open-source-catalogs) [Feature Comparison](https://www.ryft.io/blog/how-to-choose-an-apache-iceberg-catalog#feature-comparison) [Wrapping Up](https://www.ryft.io/blog/how-to-choose-an-apache-iceberg-catalog#wrapping-up) ## Browse other blogs ![Apache Iceberg V3: Is It Ready?](https://cdn.prod.website-files.com/68ef977c5ec5483ef208e8b3/69415e4b45acad26189a17d2_Blog%20Coverv3.png) Engineering ## Apache Iceberg V3: Is It Ready? Apache Iceberg V3 is a huge step forward for the lakehouse ecosystem. The V3 specification was finalized and ratified earlier this year, bringing several long-awaited capabilities into the core of the format: efficient row-level deletes, built-in row lineage, better handling of semi-structured data, and the beginnings of native encryption. This post breaks down the major features, the current state of implementation, and what this means for real adoption. ![](https://cdn.prod.website-files.com/68ef977c5ec5483ef208e8b3/68f1061b22ca6c79303ddfab_Auteur_pic_01%20(3).avif) Guy Yasoor December 16, 2025 December 17, 2025 [blog](https://www.ryft.io/blog/apache-iceberg-v3-is-it-ready) ![Streaming with Apache Iceberg: The Operational Problems at Scale](https://cdn.prod.website-files.com/68ef977c5ec5483ef208e8b3/692895683e8be6007b3e1902_Blog%20Cover.png) Engineering ## Streaming with Apache Iceberg: The Operational Problems at Scale Streaming into Iceberg creates three operational problems most teams don't see coming: small files pile up faster than you can compact them, storage costs climb because you're paying for data you've already replaced, and merges take much longer than they should. After seeing these problems in real production data lakes, we decided to share more about the causes and possible solutions. ![](https://cdn.prod.website-files.com/68ef977c5ec5483ef208e8b3/68f105fe51b1bd3a206f8f26_Auteur_pic_01%20(2).avif) Yuval Yogev December 8, 2025 December 1, 2025 [blog](https://www.ryft.io/blog/streaming-with-apache-iceberg-the-operational-problems-at-scale) ![How to Choose an Apache Iceberg Catalog](https://cdn.prod.website-files.com/68ef977c5ec5483ef208e8b3/692068eb2d13507cc7245b1e_Blog%20Cover.png) Engineering ## How to Choose an Apache Iceberg Catalog Apache Iceberg has become the table format of choice for building open data lakehouses. It solves long-standing problems around ACID transactions and engine interoperability. This post covers how to choose an Iceberg Catalog for production. ![](https://cdn.prod.website-files.com/68ef977c5ec5483ef208e8b3/68f1061b22ca6c79303ddfab_Auteur_pic_01%20(3).avif) Guy Yasoor December 8, 2025 November 21, 2025 [blog](https://www.ryft.io/blog/how-to-choose-an-apache-iceberg-catalog) ## See Ryft in Action The only solution that automatically maintains and optimizes your Iceberg tables based on usage, no lock-in, no manual tuning [See Ryft in Action](https://www.ryft.io/see-ryft-in-action) ![screening illustration](https://cdn.prod.website-files.com/68cf7feaa53a47caf3256f08/68ee7e458fe1bc5e963b6380_Frame%201707484444.avif) ![blog cta ](https://cdn.prod.website-files.com/68cf7feaa53a47caf3256f08/68efc958cdd00391780e03c2_Frame%201707484444%20(1).avif) [home](https://www.ryft.io/) ## Unlocking Iceberg Management [Ryft Blog](https://www.ryft.io/blog) News # Unlocking Iceberg management for everyone ![](https://cdn.prod.website-files.com/68ef977c5ec5483ef208e8b3/68f105da696b0f9017b6c9ed_Auteur_pic_01.avif) Yossi Reitblat July 8, 2025 ![](https://cdn.prod.website-files.com/68ef977c5ec5483ef208e8b3/68efac34c4925bbf47b6a113_689363b8bb6dd97ccd38b882_686d09873da2afe9164ed0ba_cover.avif) ### **Ryft is officially out of stealth - Unlocking Iceberg management for everyone** Today, I’m excited to announce that Ryft is coming out of stealth to enable every company in the world true control and ownership over their data. **15 years in the making** Ryft, in many ways, is a story 15 years in the making. Yuval Yogev, Guy Gadon and I went to the same high school, worked together at 8200, building high-scale data infrastructure, and went our separate ways - all to realize that we really enjoy solving complicated data infrastructure problems together with the people we love. Through our shared experiences, building complex infrastructure at Meta, scaling PB-scale data platforms at Sygnia, and building consumer products used by millions of people, we’ve set out on a journey to bring simplicity to one of the most complicated areas in data infrastructure: open data platforms. **Apache Iceberg & AI = a perfect storm** In the age of AI, data has become the most important asset we have. The ability to move fast and innovate is directly influenced by our flexibility and agility - being able to utilize data in whatever way we want, and do so with no constraints, sets the basis for our entire data operation. Apache Iceberg opens the door to something that was previously only possible for a select few companies - controlling and owning your data, without sacrificing performance and critical database capabilities. But with great power (over your data), comes… well, you know. Companies now find themselves having to choose between two far-from-ideal options: a vendor-locked solution, with no flexibility and control, or, a complicated, error-prone and manual open platform. But what if you could get the best of both worlds? Being able to switch from one platform to another with little overhead, ingesting and querying with any tool you want, and even utilizing different cloud providers, while still having your data secure, compliant, in top-performance shape and most importantly **fully-controlled by you**? This is exactly what we set out to do. **Ryft sets out to end this dilemma** We've built the best Iceberg Management platform in the world. Focused on solving the hardest problems with managing storage and data - making the storage as efficient as possible, storing data based on access patterns to reduce data scans, dealing with compliance, data lifecycle and disaster recovery at scale. Our goal with Ryft is to allow any company in the world to execute and innovate faster with their data. No company should be limited by high data infrastructure costs, or have barriers in accessing their data - the future of data infrastructure is open and in control, and we are going to lead the way. Over the past 10 months, we’ve had the honor to work with some of the best people we know, and it’s clear to me that the only way to build incredible software is to work with talented, fun and passionate people. So to our entire team: we are just getting started, and we couldn’t have done it, and continue to do it, without you. Stay tuned. It’s only the beginning. Table of Contents [Ryft is officially out of stealth - Unlocking Iceberg management for everyone](https://www.ryft.io/blog/unlocking-iceberg-management-for-everyone#ryft-is-officially-out-of-stealth-unlocking-iceberg-management-for-everyone) Get the latest posts straight to your inbox Thank you! Your submission has been received! ## Browse other blogs ![Announcing Ryft Data Retention & Compliance Enforcement for Apache Iceberg](https://cdn.prod.website-files.com/68ef977c5ec5483ef208e8b3/6936da25feec7bdfd478e008_Blog%20Cover.png) News ## Announcing Ryft Data Retention & Compliance Enforcement for Apache Iceberg Today, we’re introducing two new capabilities in Ryft: Automated Data Retention and Data Compliance Enforcement for Apache Iceberg™. These features integrate directly into the Ryft platform to ensure efficient, policy-driven data deletion and compliance, working seamlessly alongside table maintenance and optimization. ![](https://cdn.prod.website-files.com/68ef977c5ec5483ef208e8b3/68f105fe51b1bd3a206f8f26_Auteur_pic_01%20(2).avif) Yuval Yogev December 8, 2025 December 8, 2025 [blog](https://www.ryft.io/blog/announcing-ryft-data-retention-compliance-enforcement-for-apache-iceberg) ![Announcing Ryft Adaptive Optimization](https://cdn.prod.website-files.com/68ef977c5ec5483ef208e8b3/68efaaa4554eca8245e5f2f2_68caa4aa095abc61c9362c0b_Blog%2520Cover%2520(1).avif) News ## Announcing Ryft Adaptive Optimization Today, we’re officially introducing Ryft Adaptive Optimization - always-on, dynamic optimization engine for Apache Iceberg™. Our engine continuously compacts, rewrites, indexes, and reorders data based on how your tables are actually used, delivering up to 5× faster queries, 10x storage reduction, and 7x better compaction efficiency compared to other engines. ![](https://cdn.prod.website-files.com/68ef977c5ec5483ef208e8b3/68f105da696b0f9017b6c9ed_Auteur_pic_01.avif) Yossi Reitblat October 23, 2025 September 17, 2025 [blog](https://www.ryft.io/blog/announcing-ryft-adaptive-optimization) ![Unlocking Iceberg management for everyone](https://cdn.prod.website-files.com/68ef977c5ec5483ef208e8b3/68efac34c4925bbf47b6a113_689363b8bb6dd97ccd38b882_686d09873da2afe9164ed0ba_cover.avif) News ## Unlocking Iceberg management for everyone Ryft, in many ways, is a story 15 years in the making. Yuval Yogev, Guy Gadon and I went to the same high school, worked together at 8200, building high-scale data infrastructure, and went our separate ways - all to realize that we really enjoy solving complicated data infrastructure problems together with the people we love. ![](https://cdn.prod.website-files.com/68ef977c5ec5483ef208e8b3/68f105da696b0f9017b6c9ed_Auteur_pic_01.avif) Yossi Reitblat October 27, 2025 July 8, 2025 [blog](https://www.ryft.io/blog/unlocking-iceberg-management-for-everyone) ## See Ryft in Action The only solution that automatically maintains and optimizes your Iceberg tables based on usage, no lock-in, no manual tuning [See Ryft in Action](https://www.ryft.io/see-ryft-in-action) ![screening illustration](https://cdn.prod.website-files.com/68cf7feaa53a47caf3256f08/68ee7e458fe1bc5e963b6380_Frame%201707484444.avif) ![blog cta ](https://cdn.prod.website-files.com/68cf7feaa53a47caf3256f08/68efc958cdd00391780e03c2_Frame%201707484444%20(1).avif) [home](https://www.ryft.io/) ## Apache Iceberg Statistics Overview [Ryft Blog](https://www.ryft.io/blog) Engineering # Making Sense of Apache Iceberg Statistics ![](https://cdn.prod.website-files.com/68ef977c5ec5483ef208e8b3/68f1061b22ca6c79303ddfab_Auteur_pic_01%20(3).avif) Guy Yasoor May 26, 2025 ![](https://cdn.prod.website-files.com/68ef977c5ec5483ef208e8b3/68efac2e5e21b981698a1e96_68403bf334ef9009d11fae53_683b6409f6c82555d8e9b635_cover%2520post%2520(1).avif) Apache Iceberg™ is known for its rich metadata model, and one of its most powerful (but often confusing) features is its support for statistics. These statistics power everything from query planning to table maintenance and optimization, but they live across multiple layers: files, manifests, partitions, and snapshots. In this blog post we will break them down, helping you understand what exists today, what you should configure, and what’s coming next. ## Why Statistics Matter in Iceberg Iceberg collects statistics to avoid reading unnecessary data — a crucial performance feature in large-scale analytics. Instead of scanning entire files blindly, query engines can inspect metadata like column min/max values, partition-level record counts, or manifest summaries to skip irrelevant data. The result: reduced I/O, faster queries, and lower compute costs. We’ve found it helps to think of Iceberg stats in two groups: data-level (what’s inside the files) and metadata-level (how the files are organized). Keeping this distinction in mind makes it easier to understand their impact. We’ll now break down each group to see how these stats work and where they matter. ### Part 1: Data-Level Statistics These statistics describe the **actual contents** of data files. They’re essential for query planning and pruning. #### Parquet Row Group Statistics - **Source:** Stored in Parquet row groups. - **Includes:** min/max, null/NaN count per column per row group. - **Used for:** Predicate pushdown and fine-grained file skipping. - **Enabled by default?** Yes, used in most writers. - **Reference:** [Parquet format spec](https://github.com/apache/parquet-format) - **Note:** Parquet collects statistics for many different levels of its structure (chunks, pages, etc.), but we will not go into all of them in this post. #### Data File Statistics - **Source:** Stored in manifest entries for each data file. - **Includes:** Aggregated stats for each column in the data file (value count, min/max, null/NaN count). - **Used for:** File-level filtering and scan planning. - **Enabled by default?** Yes, but only on the first 100 columns. This can be changed by configuring the table properties `write.metadata.metrics.default` or `write.metadata.metrics.max-inferred-column-defaults`. - **Reference:** [Iceberg Spec — Manifests](https://iceberg.apache.org/spec/#manifests) #### Bloom Filters - **Source:** Stored in Parquet and ORC files, when enabled. - **Includes:** Per column, indicates whether it’s definitely not in the file, or it might be in the file. - **Used for:** Fast filtering on high-cardinality columns (e.g. `user_id`, `email`). Bloom filters help quickly eliminate files where a value definitely doesn't exist. This is especially effective when scanning large datasets with selective equality filters. - **Enabled by default?** No. Supported by most writers when configuring the table properties `write.parquet.bloom-filter-enabled.column.` or `write.orc.bloom.filter.columns` - Reference: [Parquet bloom filter spec](https://github.com/apache/parquet-format/blob/master/BloomFilter.md) #### Table Statistics (NDV Puffin Files) - **Source:** Blobs stored in Puffin metadata files. - **Includes:** Theta sketch — a probabilistic data structure for estimating NDV (number of distinct values) for each column. - **Used for:** Join planning, estimating cardinality, improving cost-based optimizer decisions. - **Enabled by default?** No. Some query engines can utilize Puffin files for read optimizations, but writing is still not widely adopted and is not fully standardized — supported only in some query engines such as Trino, or in Spark via the [compute\_table\_stats](https://iceberg.apache.org/docs/nightly/spark-procedures/#compute_table_stats) procedure. - **Reference:** [Puffin format spec](https://iceberg.apache.org/puffin-spec/) ### Part 2: Metadata-Level Statistics These don’t describe the data itself — they summarize **table structure and file layout**. These are easy to confuse with data-level stats but serve a different purpose. #### Partition Statistics - **What they track:** Record counts and file counts for each partition. (Proposal [#11083](https://github.com/apache/iceberg/issues/11083) would allow collecting **column-level stats per partition**, making this also a data-level statistics file) - **Used for:** Estimating data distribution. - **Enabled by default?** No. This feature is not yet widely adopted, and isn’t used by most writers. - **Reference:** [Partition statistics](https://iceberg.apache.org/spec/#partition-statistics) #### Manifest List Statistics - **What they track:** Number of added, existing, and deleted files for the snapshot; summary statistics for each partition (min/max, null/NaN count). - **Used for:** Partition pruning, and skipping entire manifest files during planning. - **Enabled by default?** Yes. - **Reference:** [Manifest list spec](https://iceberg.apache.org/spec/#manifest-lists) #### Snapshot Summary Fields - **What they track:** Number of files added, deleted; total records affected. - **Used for:** Auditing and planning. - **Enabled by default?** Yes, these are optional fields but are written by most query engines. - **Reference:** [Snapshot summary fields](https://iceberg.apache.org/spec/#optional-snapshot-summary-fields) ## Summary Apache Iceberg’s magic lies in its rich metadata layer, and statistics are a big part of that. Each layer of stats serves a different purpose. Parquet stats and manifest stats enable fast filtering, Bloom filters speed up scans on high-cardinality columns, Puffin improves join planning, and metadata stats enable better pruning and planning. Some of these features are off by default, or require tuning to provide the best performance boost. We’ve prepared a handy cheat-sheet so you don’t have to remember this every time: ![Iceberg statistics](https://cdn.prod.website-files.com/68ef977c5ec5483ef208e8b3/68efac2d5e21b981698a1e8b_683b639170aaab3811c900c2_1*LdklTkzSgMU52fefdpmu2w.avif) Apache Iceberg ![Iceberg statistics summary](https://cdn.prod.website-files.com/68ef977c5ec5483ef208e8b3/68efac2d5e21b981698a1e87_683b6391e0b90cfb6b924aa1_1*Vd9p7Hy63QMP42F9mPzV4Q.avif) ## Final Thoughts There’s a lot going on under the hood with Iceberg stats, but it’s not just complexity for the sake of complexity. Each layer plays a role, some help skip files, others help join planning or speedup scans. You don’t have to flip every switch on day one, but knowing what’s there lets you be smarter about tuning things as your workloads grow. Start with the defaults, and layer in more as needed. As new features roll out — like per-partition column stats — things will keep improving, but they’ll also add complexity. We’re already testing them in real-world environments and will keep sharing what we learn. Got questions or ideas? Reach out at [hi@ryft.io](mailto:hi@ryft.io) Table of Contents [Why Statistics Matter in Iceberg](https://www.ryft.io/blog/making-sense-of-apache-iceberg-statistics#why-statistics-matter-in-iceberg) [Part 1: Data-Level Statistics](https://www.ryft.io/blog/making-sense-of-apache-iceberg-statistics#part-1-data-level-statistics) [Part 2: Metadata-Level Statistics](https://www.ryft.io/blog/making-sense-of-apache-iceberg-statistics#part-2-metadata-level-statistics) [Summary](https://www.ryft.io/blog/making-sense-of-apache-iceberg-statistics#summary) [Final Thoughts](https://www.ryft.io/blog/making-sense-of-apache-iceberg-statistics#final-thoughts) Get the latest posts straight to your inbox Thank you! Your submission has been received! ## Browse other blogs ![Apache Iceberg V3: Is It Ready?](https://cdn.prod.website-files.com/68ef977c5ec5483ef208e8b3/69415e4b45acad26189a17d2_Blog%20Coverv3.png) Engineering ## Apache Iceberg V3: Is It Ready? Apache Iceberg V3 is a huge step forward for the lakehouse ecosystem. The V3 specification was finalized and ratified earlier this year, bringing several long-awaited capabilities into the core of the format: efficient row-level deletes, built-in row lineage, better handling of semi-structured data, and the beginnings of native encryption. This post breaks down the major features, the current state of implementation, and what this means for real adoption. ![](https://cdn.prod.website-files.com/68ef977c5ec5483ef208e8b3/68f1061b22ca6c79303ddfab_Auteur_pic_01%20(3).avif) Guy Yasoor December 16, 2025 December 17, 2025 [blog](https://www.ryft.io/blog/apache-iceberg-v3-is-it-ready) ![Streaming with Apache Iceberg: The Operational Problems at Scale](https://cdn.prod.website-files.com/68ef977c5ec5483ef208e8b3/692895683e8be6007b3e1902_Blog%20Cover.png) Engineering ## Streaming with Apache Iceberg: The Operational Problems at Scale Streaming into Iceberg creates three operational problems most teams don't see coming: small files pile up faster than you can compact them, storage costs climb because you're paying for data you've already replaced, and merges take much longer than they should. After seeing these problems in real production data lakes, we decided to share more about the causes and possible solutions. ![](https://cdn.prod.website-files.com/68ef977c5ec5483ef208e8b3/68f105fe51b1bd3a206f8f26_Auteur_pic_01%20(2).avif) Yuval Yogev December 8, 2025 December 1, 2025 [blog](https://www.ryft.io/blog/streaming-with-apache-iceberg-the-operational-problems-at-scale) ![How to Choose an Apache Iceberg Catalog](https://cdn.prod.website-files.com/68ef977c5ec5483ef208e8b3/692068eb2d13507cc7245b1e_Blog%20Cover.png) Engineering ## How to Choose an Apache Iceberg Catalog Apache Iceberg has become the table format of choice for building open data lakehouses. It solves long-standing problems around ACID transactions and engine interoperability. This post covers how to choose an Iceberg Catalog for production. ![](https://cdn.prod.website-files.com/68ef977c5ec5483ef208e8b3/68f1061b22ca6c79303ddfab_Auteur_pic_01%20(3).avif) Guy Yasoor December 8, 2025 November 21, 2025 [blog](https://www.ryft.io/blog/how-to-choose-an-apache-iceberg-catalog) ## See Ryft in Action The only solution that automatically maintains and optimizes your Iceberg tables based on usage, no lock-in, no manual tuning [See Ryft in Action](https://www.ryft.io/see-ryft-in-action) ![screening illustration](https://cdn.prod.website-files.com/68cf7feaa53a47caf3256f08/68ee7e458fe1bc5e963b6380_Frame%201707484444.avif) ![blog cta ](https://cdn.prod.website-files.com/68cf7feaa53a47caf3256f08/68efc958cdd00391780e03c2_Frame%201707484444%20(1).avif) [home](https://www.ryft.io/) ## Athena vs. Snowflake Comparison [Ryft Blog](https://www.ryft.io/blog) Engineering # Athena vs. Snowflake on Iceberg: Performance and Cost Comparison on TPC-H ![](https://cdn.prod.website-files.com/68ef977c5ec5483ef208e8b3/68f105fe51b1bd3a206f8f26_Auteur_pic_01%20(2).avif) ![](https://cdn.prod.website-files.com/68ef977c5ec5483ef208e8b3/68f105da696b0f9017b6c9ed_Auteur_pic_01.avif) Yuval Yogev Yossi Reitblat May 26, 2025 ![](https://cdn.prod.website-files.com/68ef977c5ec5483ef208e8b3/68efaaa40baa4b79fdea5613_689363feed050d2145cbe300_68403b6d7aafbecebd5589e2_683b6564cd8fe3c5baf29a65_Option%25201.avif) ### Preface At Ryft, we’re constantly testing different query engines, layouts, and configs to bring the best performance out of lakehouses. One of the great things about **Iceberg** as an open table format is that it lets us compare engines objectively, since they operate on the **same single copy of the data**. This eliminates differences in storage formats and data duplication, making the results more meaningful. When picking a query engine, **Amazon Athena** and **Snowflake** are two popular names that always come up. Both are cloud-based, support SQL, and play nicely with modern data stacks. But how do they compare in terms of performance and cost?We ran a TPC-H benchmark on both systems to find out. Before diving into the results, a few disclaimers: **Benchmarking is hard.** It’s not just about running a few queries and comparing numbers — **tiny differences in setup, data layout, or query optimizations can completely change the results.** Performance depends on workload, data structure, concurrency, and how well each system is tuned. The numbers we’re sharing reflect **our specific setup**, not some universal truth. Think of this as a **real-world comparison**, not a final verdict on which engine is “better.” **Architectural differences matter.** Snowflake and Athena have fundamentally different designs that impact cost and scalability. Snowflake runs on virtual warehouses, where you provision dedicated compute resources and manually scale them up or down. This allows for better performance control but requires active management. Athena, being serverless, charges per TB of data scanned and doesn’t let you adjust compute resources directly, making it simpler but limiting tuning options. **Performance isn’t everything.** It’s easy to get caught up in benchmark numbers, but real-world decisions aren’t just about raw speed.Cost, ease of use, ecosystem integration, and day-to-day operational complexity all matter just as much — sometimes even more. There’s a great blog post from the folks at MotherDuck, [_“Perf is not enough,”_](https://motherduck.com/blog/perf-is-not-enough/) — They make a solid point: the fastest system isn’t always the best one. At the end of the day, the best tool is the one that fits your workload, your team, and your budget — **without making your life harder.** ### Benchmark Setup For fairness, we ran the same TPC-H dataset on both engines, stored in **Apache Iceberg** format. Iceberg is a modern table format that supports ACID transactions and efficient data pruning — features that both Athena and Snowflake can leverage. #### Configurations - **Athena**: We used **Athena SQL engine version 3** (powered by Trino). - **Snowflake**: We ran queries on a **Small** **&** **Medium virtual warehouse.** - **Data:** The data was generated using [DuckDB TPC-H extension](https://duckdb.org/docs/extensions/tpch.html) with a scale factor of 1024 which results with roughly **~10 billion records** in total, and **~350GB** compressed data size, **~1.7TB** uncompressed, saved on S3. - **Data Format**: All tables were stored in **Iceberg with Parquet file and zstd compression** to ensure that both engines had access to the same performance optimizations. - **Data Layout**: TPC-H does not determine how data should be partitioned or clustered, which as stated before can drastically change benchmark results. In our dataset, the biggest tables were partitioned by time. Monthly partitioning was applied on **`orders`** and **`lineitem`** tables, by **`o_orderdate`,** and **`(l_shipdate, l_commitdate)`** respectively. The data layout is far from being optimal, but it is consistent for this benchmark. ### Performance Comparison ![Snowflake vs Athena performance comparison](https://cdn.prod.website-files.com/68ef977c5ec5483ef208e8b3/68efaaa40baa4b79fdea55ef_683b64f70111645bd16dcb1e_1*jKvCfvx7X7bWga-StpUseQ.avif) Runtime is in seconds ![Snowflake vs Athena performance comparison](https://cdn.prod.website-files.com/68ef977c5ec5483ef208e8b3/68efaaa40baa4b79fdea55ec_683b64f6daf08e0f183bd976_1*zNGOMSIKwRbwsUUJJ9KI6A.avif) ### Cost Comparison Cost comparison was done using the following parameters: - 1 Credit = 3.00$ (Enterprise edition cost) - Small Warehouse = 0.0006 credits per second - Medium Warehouse = 0.0011 credits per second - Athena scan cost = 0.005 per GB scanned ![Snowflake vs Athena cost comparison](https://cdn.prod.website-files.com/68ef977c5ec5483ef208e8b3/68efaaa40baa4b79fdea55f2_683b64f68bdb48594a1a50f8_1*EskPDFyPRcPwrkXoOit1FA.avif) ![Snowflake vs Athena cost comparison](https://cdn.prod.website-files.com/68ef977c5ec5483ef208e8b3/68efaaa40baa4b79fdea55e8_683b64f6cd8fe3c5baf261a2_1*eEL2i7tv2qINz2BebTFnAg.avif) _Note: Calculating “cost per query” in snowflake is not trivial, as warehouses can run multiple queries at once, and also can suspend and resume based on activity._ - Athena Total Cost: **$4.62** - Snowflake Small Total Cost: **$2.65** - Snowflake Medium Total Cost: **$2.38** ### Key Observations Athena was faster in **8 out of 22 queries**, while Snowflake was **cheaper in 18 out of 22**. Overall, **Snowflake’s total cost was 49% lower** (not including S3 costs, which apply to both engines). Interestingly, opting for a **more expensive Snowflake warehouse didn’t increase overall costs** — it actually **reduced them**. Higher-tier warehouses helped eliminate bottlenecks like disk spillage from low-memory configurations, leading to better efficiency. ### Final Thoughts The most exciting part of this comparison isn’t just the results — it’s that we could benchmark two different query engines on the exact same Iceberg dataset. Both Athena and Snowflake performed well, each excelling in different scenarios. Iceberg’s support in both engines was solid, proving its potential as an open table format for analytics at scale. That said, there’s still room for improvement, especially in areas like query pruning and pushdown filters. As Iceberg matures, we can expect even better performance and efficiency, making it an even more compelling choice for modern data architectures. _Stay tuned for more benchmarks in the future, and feel free to reach out if there’s an engine or data layout you’d like us to test next!_ Table of Contents [Preface](https://www.ryft.io/blog/athena-vs-snowflake-on-iceberg-tpch-comparison#preface) [Benchmark Setup](https://www.ryft.io/blog/athena-vs-snowflake-on-iceberg-tpch-comparison#benchmark-setup) [Performance Comparison](https://www.ryft.io/blog/athena-vs-snowflake-on-iceberg-tpch-comparison#performance-comparison) [Cost Comparison](https://www.ryft.io/blog/athena-vs-snowflake-on-iceberg-tpch-comparison#cost-comparison) [Key Observations](https://www.ryft.io/blog/athena-vs-snowflake-on-iceberg-tpch-comparison#key-observations) [Final Thoughts](https://www.ryft.io/blog/athena-vs-snowflake-on-iceberg-tpch-comparison#final-thoughts) Get the latest posts straight to your inbox Thank you! Your submission has been received! ## Browse other blogs ![Apache Iceberg V3: Is It Ready?](https://cdn.prod.website-files.com/68ef977c5ec5483ef208e8b3/69415e4b45acad26189a17d2_Blog%20Coverv3.png) Engineering ## Apache Iceberg V3: Is It Ready? Apache Iceberg V3 is a huge step forward for the lakehouse ecosystem. The V3 specification was finalized and ratified earlier this year, bringing several long-awaited capabilities into the core of the format: efficient row-level deletes, built-in row lineage, better handling of semi-structured data, and the beginnings of native encryption. This post breaks down the major features, the current state of implementation, and what this means for real adoption. ![](https://cdn.prod.website-files.com/68ef977c5ec5483ef208e8b3/68f1061b22ca6c79303ddfab_Auteur_pic_01%20(3).avif) Guy Yasoor December 16, 2025 December 17, 2025 [blog](https://www.ryft.io/blog/apache-iceberg-v3-is-it-ready) ![Streaming with Apache Iceberg: The Operational Problems at Scale](https://cdn.prod.website-files.com/68ef977c5ec5483ef208e8b3/692895683e8be6007b3e1902_Blog%20Cover.png) Engineering ## Streaming with Apache Iceberg: The Operational Problems at Scale Streaming into Iceberg creates three operational problems most teams don't see coming: small files pile up faster than you can compact them, storage costs climb because you're paying for data you've already replaced, and merges take much longer than they should. After seeing these problems in real production data lakes, we decided to share more about the causes and possible solutions. ![](https://cdn.prod.website-files.com/68ef977c5ec5483ef208e8b3/68f105fe51b1bd3a206f8f26_Auteur_pic_01%20(2).avif) Yuval Yogev December 8, 2025 December 1, 2025 [blog](https://www.ryft.io/blog/streaming-with-apache-iceberg-the-operational-problems-at-scale) ![How to Choose an Apache Iceberg Catalog](https://cdn.prod.website-files.com/68ef977c5ec5483ef208e8b3/692068eb2d13507cc7245b1e_Blog%20Cover.png) Engineering ## How to Choose an Apache Iceberg Catalog Apache Iceberg has become the table format of choice for building open data lakehouses. It solves long-standing problems around ACID transactions and engine interoperability. This post covers how to choose an Iceberg Catalog for production. ![](https://cdn.prod.website-files.com/68ef977c5ec5483ef208e8b3/68f1061b22ca6c79303ddfab_Auteur_pic_01%20(3).avif) Guy Yasoor December 8, 2025 November 21, 2025 [blog](https://www.ryft.io/blog/how-to-choose-an-apache-iceberg-catalog) ## See Ryft in Action The only solution that automatically maintains and optimizes your Iceberg tables based on usage, no lock-in, no manual tuning [See Ryft in Action](https://www.ryft.io/see-ryft-in-action) ![screening illustration](https://cdn.prod.website-files.com/68cf7feaa53a47caf3256f08/68ee7e458fe1bc5e963b6380_Frame%201707484444.avif) ![blog cta ](https://cdn.prod.website-files.com/68cf7feaa53a47caf3256f08/68efc958cdd00391780e03c2_Frame%201707484444%20(1).avif) [home](https://www.ryft.io/) ## Unlocking Security Data Lake [Ryft Blog](https://www.ryft.io/blog) Engineering # Why Apache Iceberg Finally Unlocks Security Data Lakes ![](https://cdn.prod.website-files.com/68ef977c5ec5483ef208e8b3/68f105fe51b1bd3a206f8f26_Auteur_pic_01%20(2).avif) ![](https://cdn.prod.website-files.com/68ef977c5ec5483ef208e8b3/68f1061b22ca6c79303ddfab_Auteur_pic_01%20(3).avif) Yuval Yogev Guy Yasoor August 6, 2025 ![](https://cdn.prod.website-files.com/68ef977c5ec5483ef208e8b3/68efaaad2cfdee77c8cfca4d_68934f1203ec38af84c342dc_cyber_blog_cover.avif) Security is actually a Big Data problem. If you could store all the data in the world, query it in sub-second speed, model entities correctly and correlate all data sources, it would be a huge step forward in detecting and preventing cyber attacks. It’s no surprise that [literally](https://www.microsoft.com/en-us/security/blog/2025/07/22/microsoft-sentinel-data-lake-unify-signals-cut-costs-and-power-agentic-ai/) [everyone](https://aws.amazon.com/security-lake/) is trying to [build](https://www.detectionatscale.com/p/the-transition-from-monolithic-siems) a [security data lake](https://graylog.org/post/security-data-lake-strategy/). However, security data lakes are notoriously [hard](https://medium.com/anton-on-security/why-your-security-data-lake-project-will-well-actually-78e0e360c292) to build and operate efficiently. #### **The nature of security workloads** Security data infrastructure is fundamentally different from classical analytics. It needs to support: - **Always-on detections** - **Deep investigations** - **Massive data volumes** - **Complex joins across sources** - **Constantly changing schemas** - **Strict compliance & data sovereignty** - **AI-driven enrichment and modeling** **Security Teams** need to keep years of logs searchable at low cost, while running everything from threat hunting to compliance audits. **Security Vendors** need to offer all of the above - at scale, for multiple customers - while maintaining healthy margins and avoiding lock-in. The first generation of security data lakes usually relied on tools like Splunk, Elasticsearch, or other NoSQL databases. Then, solutions like Snowflake and Databricks made a big step forward in flexibility and architecture, and became popular among security teams. However, all of these proved to be either too expensive to store raw data long-term, too inflexible to support varied workloads, or too tightly coupled to a single vendor. For vendors providing offerings like SIEM, XDR and others, this proved to be a good way to start, but a challenging approach long term, as margins became almost non-existent. Apache Iceberg changes the equation. It’s an open table format purpose-built for scalable, flexible, and cost-effective analytics - and it’s quickly becoming the new standard for modern cybersecurity data lakes. **Here’s why.** ## Future-proof infrastructure: open, scalable, efficient ### Unlimited retention at lower costs Storing years of raw security telemetry is often impossible with traditional architectures. Costs skyrocket. Detection and investigation capabilities are reduced with shrinking retention windows. Talking with security teams and vendors, you often hear sentences like: - **_“When we ingested raw CloudTrail logs into Elasticsearch without filtering, we ran out of storage in weeks”_** - **_“We only retain the EDR alerts, and not the EDR raw events. We know it could be useful for incident response, but we couldn’t find the budget”_** Iceberg separates compute from storage (for real this time), and is designed to operate directly on cheap object storage like S3. For folks that get special AWS discounts this is amplified as they can apply those on their storage directly, without a margin from a data warehouse in the middle. Of course, even S3 can become very expensive at large scale, however with Iceberg the ability to control it is higher, from storage format tuning, to tiering and retention. ### **Better margins for security vendors** For security vendors, whether you're building a detection and response platform, or a SIEM, infrastructure efficiency directly impacts your margins. Iceberg helps by letting you scale compute independently from storage. You can run lightweight engines for real-time detections, and spin up heavier compute only when needed for deep investigations or model training. This architecture avoids the overhead of always-on clusters and eliminates the need to pass on bloated infrastructure costs to customers. You’re no longer forced to rely on upstream vendors and their pricing models just to make your own offering viable. ### **No vendor lock-in** Iceberg is fully open - no proprietary formats, no vendor-specific runtimes. You control where it runs (your cloud, your region), and which engines you use to query it. This flexibility is critical for any vendor or security team that wants to stay in control of its data and tooling. Your ability to make experiments, use the newest tools in the market, and at the end of the day make the most of your data, unlocks instantly. More on that in the next section. ### Data ownership When customer data is sensitive, and in security, it always is - you can’t afford to ship it to a black-box SaaS. Iceberg allows for flexible deployments, so you can more easily support air-gapped, on-prem or BYOC deployments. Even more importantly, you no longer have to send your customers’ most sensitive data to a third-party vendor and trust they’ll treat it with the same care you would. ## **One table format, many engines** ### **Choosing the best tool for the job** Security teams don’t just run one type of workload - they run many. From continuous detections, to point lookups, cross correlation of sources, IOC searches, AI model training and more. If you found a data solution that supports all those in one place, that would be amazing, but until then, Iceberg unlocks the ability to choose the best tool for each workload. Imagine if you could use the following tools on your authentication logs: - **ClickHouse** for dashboards identifying trends like new user-agents and geo-locations - **Spark** for streaming detections, and enrichment with external data sources like GeoIP databases - **Trino** for IOC sweeps and other ad-hoc queries - **Graph engines like PuppyGraph** for complex detections like impossible travel and lateral movement - **Python/DuckDB** for quick pivots and prototyping All of these engines can query the same Iceberg tables - directly. With Iceberg, you don’t need to maintain multiple pipelines or duplicate data across formats. Data is written once and read by any engine, using standard table APIs. That means less infrastructure, less complexity, and far faster iteration across teams. ![One Table Format, Many Engines](https://cdn.prod.website-files.com/68ef977c5ec5483ef208e8b3/68efaaac2cfdee77c8cfc9cf_68934ffd4d449c491fa0950b_cyber_lake.avif) ### Security ❤️ AI Every security team and vendor today is trying to figure out how to 10x their capabilities with AI and LLMs - whether it’s automated triage, summarizing alerts, or querying telemetry with natural language. But if your data is locked inside a vendor platform, you're stuck. You either wait for them to build exactly what you need, or start copying data into yet another system just to make it usable. With Iceberg, your data is immediately accessible to modern AI tooling. You can point tools like **LangChain** or **LlamaIndex** at your tables, or train your own models on raw logs using **Spark,** **Ray** or **Daft,** all without exporting or transforming the data. ## **Iceberg features that fit cybersecurity workloads** ### **Schema evolution** Security data changes constantly. New sources, new fields, new vendors - and standards like OCSF are evolving quickly. Iceberg supports field-level schema evolution. You can add, rename, or reorder fields without breaking existing queries or reprocessing old data. ### **Partition evolution** You can also evolve how tables are partitioned over time. Start with timestamp-based partitions, then add dimensions like **_`event_type`_** or **_`customer_id`_** as access patterns evolve. No table rewrites required. With advanced optimizations like adaptive partitioning, you could even change your partitioning strategy dynamically. For example: CloudTrail logs might require hourly partitions during peak working hours, but can be switched to daily partitions on weekends. ### **Time travel and snapshot isolation** Iceberg supports snapshot-based queries, so you can “go back in time” and rerun detection logic exactly as it would have worked during a breach. Or reconstruct the full system state for a post-incident report. Or compare the before-and-after of a dataset modified during an IR process. ### **Merge support for enrichment and tagging** Security data isn’t static, and you often need to tag historical records (e.g. "this IP was malicious") or apply enrichment after ingestion. Iceberg supports upserts and row-level deletes, so you can safely update your data without full rewrites. ## Vendor adoption Vendors have already identified the opportunity of building security data lakes on top of Apache Iceberg. **Amazon Security Lake** was announced with native support for Apache Iceberg, which set the standard for other parts of the ecosystem, enabling dozens of partners to ingest data directly in Iceberg format. Current integrations include **DataBahn**, **Monad**, **SailPoint**, **Sysdig**, **Talon**, **Palo Alto Networks**, **Securonix** and more. ![Vendor Adoption](https://cdn.prod.website-files.com/68ef977c5ec5483ef208e8b3/68efaaac2cfdee77c8cfc9d2_68934ebe3202024786e4bd7d_aws_cyber.avif) Ingestion tools like **Matano**, **Monad** and others, use Iceberg as their storage layer. **Crowdstrike, Orca Security** and others publicly describe using S3 + Iceberg for their workloads. **Panther Labs** supports querying Iceberg tables directly - whether sourced from Amazon Security Lake or external Snowflake-hosted Iceberg tables. [**Query.AI**](http://query.ai/) supports Iceberg tables and generates federated Athena queries over them. And **Cribl** recently launched its own Iceberg based lake product as part of their Cribl Lake offering. Additionally, dozens of security startups which are currently in stealth, are starting their data infrastructure with Iceberg as its foundation from day one. Just this week, Microsoft has joined the party and announced their [Sentinel Data Lake](https://www.microsoft.com/en-us/security/blog/2025/07/22/microsoft-sentinel-data-lake-unify-signals-cut-costs-and-power-agentic-ai/), which started making waves in the industry as well. ![Microsoft has joined the party and announced their Sentinel Data Lake, which started making waves in the industry as well.](https://cdn.prod.website-files.com/68ef977c5ec5483ef208e8b3/68efaaac2cfdee77c8cfc9d9_68934ed093d6fa07de45709c_cyber_linkedin.avif) ## **What’s coming next** Apache Iceberg is evolving fast, and it’s about to get even better for security workloads. **Coming in Iceberg V3** Native support for **semi-structured data** via the new VARIANT column type. That means queries like userIdentity.sessionContext.sessionIssuer.userName = 'John' can be queried as if userName was a top-level column, with no extra parsing or ingestion logic. **On the horizon** - Fine-grained access controls - Smarter secondary indexing - Improved support for real-time + AI-native workloads ## **Conclusion** The cybersecurity landscape is moving fast, and does not seem to slow down. Security is becoming top priority for organizations, attackers are working harder, and new AI tooling introduces novel approaches for detection and prevention. Whether you’re building a SIEM, an in-house security platform, or the next AI-native detection engine - Iceberg gives you control, flexibility, and performance that legacy solutions can’t match. **For security teams:** it’s your way to finally own your data - with performance, retention, and AI-readiness built in. **For security vendors:** it’s your infrastructure advantage - unlocking margins, features, and agility your competitors can’t replicate. We’re already seeing rapid adoption among the best security teams and vendors, building things that were unimaginable just a few years ago. Security is a big data problem. And Apache Iceberg feels like what we have been waiting for to build a truly efficient security data lake. Table of Contents [Future-proof infrastructure: open, scalable, efficient](https://www.ryft.io/blog/why-apache-iceberg-finally-unlocks-security-data-lakes#future-proof-infrastructure-open-scalable-efficient) [Unlimited retention at lower costs](https://www.ryft.io/blog/why-apache-iceberg-finally-unlocks-security-data-lakes#unlimited-retention-at-lower-costs) [Better margins for security vendors](https://www.ryft.io/blog/why-apache-iceberg-finally-unlocks-security-data-lakes#better-margins-for-security-vendors) [No vendor lock-in](https://www.ryft.io/blog/why-apache-iceberg-finally-unlocks-security-data-lakes#no-vendor-lock-in) [Data ownership](https://www.ryft.io/blog/why-apache-iceberg-finally-unlocks-security-data-lakes#data-ownership) [One table format, many engines](https://www.ryft.io/blog/why-apache-iceberg-finally-unlocks-security-data-lakes#one-table-format-many-engines) [Choosing the best tool for the job](https://www.ryft.io/blog/why-apache-iceberg-finally-unlocks-security-data-lakes#choosing-the-best-tool-for-the-job) [Security ❤️ AI](https://www.ryft.io/blog/why-apache-iceberg-finally-unlocks-security-data-lakes#security-ai) [Iceberg features that fit cybersecurity workloads](https://www.ryft.io/blog/why-apache-iceberg-finally-unlocks-security-data-lakes#iceberg-features-that-fit-cybersecurity-workloads) [Schema evolution](https://www.ryft.io/blog/why-apache-iceberg-finally-unlocks-security-data-lakes#schema-evolution) [Partition evolution](https://www.ryft.io/blog/why-apache-iceberg-finally-unlocks-security-data-lakes#partition-evolution) [Time travel and snapshot isolation](https://www.ryft.io/blog/why-apache-iceberg-finally-unlocks-security-data-lakes#time-travel-and-snapshot-isolation) [Merge support for enrichment and tagging](https://www.ryft.io/blog/why-apache-iceberg-finally-unlocks-security-data-lakes#merge-support-for-enrichment-and-tagging) [Vendor adoption](https://www.ryft.io/blog/why-apache-iceberg-finally-unlocks-security-data-lakes#vendor-adoption) [What’s coming next](https://www.ryft.io/blog/why-apache-iceberg-finally-unlocks-security-data-lakes#whats-coming-next) [Conclusion](https://www.ryft.io/blog/why-apache-iceberg-finally-unlocks-security-data-lakes#conclusion) Get the latest posts straight to your inbox Thank you! Your submission has been received! ## Browse other blogs ![Apache Iceberg V3: Is It Ready?](https://cdn.prod.website-files.com/68ef977c5ec5483ef208e8b3/69415e4b45acad26189a17d2_Blog%20Coverv3.png) Engineering ## Apache Iceberg V3: Is It Ready? Apache Iceberg V3 is a huge step forward for the lakehouse ecosystem. The V3 specification was finalized and ratified earlier this year, bringing several long-awaited capabilities into the core of the format: efficient row-level deletes, built-in row lineage, better handling of semi-structured data, and the beginnings of native encryption. This post breaks down the major features, the current state of implementation, and what this means for real adoption. ![](https://cdn.prod.website-files.com/68ef977c5ec5483ef208e8b3/68f1061b22ca6c79303ddfab_Auteur_pic_01%20(3).avif) Guy Yasoor December 16, 2025 December 17, 2025 [blog](https://www.ryft.io/blog/apache-iceberg-v3-is-it-ready) ![Streaming with Apache Iceberg: The Operational Problems at Scale](https://cdn.prod.website-files.com/68ef977c5ec5483ef208e8b3/692895683e8be6007b3e1902_Blog%20Cover.png) Engineering ## Streaming with Apache Iceberg: The Operational Problems at Scale Streaming into Iceberg creates three operational problems most teams don't see coming: small files pile up faster than you can compact them, storage costs climb because you're paying for data you've already replaced, and merges take much longer than they should. After seeing these problems in real production data lakes, we decided to share more about the causes and possible solutions. ![](https://cdn.prod.website-files.com/68ef977c5ec5483ef208e8b3/68f105fe51b1bd3a206f8f26_Auteur_pic_01%20(2).avif) Yuval Yogev December 8, 2025 December 1, 2025 [blog](https://www.ryft.io/blog/streaming-with-apache-iceberg-the-operational-problems-at-scale) ![How to Choose an Apache Iceberg Catalog](https://cdn.prod.website-files.com/68ef977c5ec5483ef208e8b3/692068eb2d13507cc7245b1e_Blog%20Cover.png) Engineering ## How to Choose an Apache Iceberg Catalog Apache Iceberg has become the table format of choice for building open data lakehouses. It solves long-standing problems around ACID transactions and engine interoperability. This post covers how to choose an Iceberg Catalog for production. ![](https://cdn.prod.website-files.com/68ef977c5ec5483ef208e8b3/68f1061b22ca6c79303ddfab_Auteur_pic_01%20(3).avif) Guy Yasoor December 8, 2025 November 21, 2025 [blog](https://www.ryft.io/blog/how-to-choose-an-apache-iceberg-catalog) ## See Ryft in Action The only solution that automatically maintains and optimizes your Iceberg tables based on usage, no lock-in, no manual tuning [See Ryft in Action](https://www.ryft.io/see-ryft-in-action) ![screening illustration](https://cdn.prod.website-files.com/68cf7feaa53a47caf3256f08/68ee7e458fe1bc5e963b6380_Frame%201707484444.avif) ![blog cta ](https://cdn.prod.website-files.com/68cf7feaa53a47caf3256f08/68efc958cdd00391780e03c2_Frame%201707484444%20(1).avif) [home](https://www.ryft.io/) ## GDPR Compliance Guide [Ryft Blog](https://www.ryft.io/blog) Engineering # GDPR Compliance with Apache Iceberg: A Practical Guide ![](https://cdn.prod.website-files.com/68ef977c5ec5483ef208e8b3/68f105da696b0f9017b6c9ed_Auteur_pic_01.avif) Yossi Reitblat July 30, 2025 ![](https://cdn.prod.website-files.com/68ef977c5ec5483ef208e8b3/68efac2e7d5065b5e6763571_688a5738f58dabd4b258cd0b_Blog%2520Cover.avif) If you've ever had to explain to a compliance officer why you can't just "delete that user's data" from your Iceberg lakehouse, you know the challenge. If you still haven’t had this conversation, this guide is for you. Iceberg's immutable architecture means that data deletion is a multi-step process rather than immediate. This requires careful operational procedures to ensure GDPR compliance, but it's completely achievable. In this guide, we're going to break down what GDPR compliance means, why it’s complicated with Iceberg, and how to build a compliant Iceberg lakehouse. There's no silver bullet, but with enough attention, it’s possible to get it right. ## **What GDPR Compliance Actually Means** GDPR compliance boils down to one critical requirement: **when a user requests deletion of their data, you must delete ALL traces of their “user identifiable information” across ALL systems and copies.** Not hide it. Not mark it as deleted. Delete it completely. **What data must be deleted:** Any information that can identify a person - names, emails, user IDs, IP addresses, device IDs, behavioral patterns, or any combination of data that could re-identify someone. **Where it must be deleted:** - Primary data - Historical snapshots and backups - Temporary or dangling files **When it must be deleted:** **** GDPR requires deletion "without undue delay" and within one month (30 days) of receiving a valid request. You can extend this to three months in complex cases, but you must inform the user within the first month. The regulation doesn't care about your storage architecture or operational complexity. If the data can be traced back to an individual, it must be gone. ## Why Is It Complicated with Iceberg? Iceberg creates three main challenges for GDPR compliance: ### 1\. Multiple Data Copies via Snapshots Iceberg maintains historical snapshots for time travel, meaning deleted data persists in old snapshots: ` ` ### 2\. User Data is Scattered Across TBs of Files and Partitions User data spreads across many files and partitions. A single user deletion might require to rewrite TBs of data. Now think doing that multiple times a day. ![](https://cdn.prod.website-files.com/68ef977c5ec5483ef208e8b3/68efac2d7d5065b5e67634d9_688a5551584040738a226f10_scattered_user_data.avif) ### 3\. Copy-on-Write vs Merge-on-Read Iceberg supports two approaches for handling deletions: 1. Copy-on-Write (COW) - immediately rewrites data files and removes deleted rows 2. Merge-on-Read (MOR) - creates separate delete files that need to be merged processed during compaction. While COW is the simpler approach and guarantees compliance for the latest snapshot, it’s not always viable - engines like Trino & Athena only support MOR writes, and sometimes the costs are too high to COW on each deletion request. ![](https://cdn.prod.website-files.com/68ef977c5ec5483ef208e8b3/68efac2d7d5065b5e67634b3_688a55b7959d7ba89d43fcbe_Copy-on-Write%2520vs%2520Merge-on-Read.avif) ## Setting Up a GDPR-Compliant Lakehouse ### Avoid the Problem If You Can The best GDPR strategy is prevention. Two approaches can eliminate most compliance complexity: **Don't Store What You Don't Need** **** Before storing any user data, ask: do we actually need this information? - Email addresses for login? Consider OAuth providers instead. - Full addresses for analytics? Geographic regions might suffice. - Individual user behavior? Aggregated cohort data might work. **Hash Identifiable Data** **** For data you must store, hash identifiable information using a secure hash function with a salt. Store the original mapping in a separate, secured table. When deletion requests arrive, delete the mapping and the hashed data becomes anonymous.Or, skip the mapping and simply hash all user identifiers to be fully compliant. **Segregate Personal Data** **** Isolate GDPR-subject data into dedicated namespaces or schemas (e.g., `sensitive.user_events` vs `analytics.aggregated_metrics`). This makes it easier to identify, manage, and delete personal data without affecting non-personal analytics tables. **The Trade-off**: You lose the ability to directly query by user ID, but you gain significant compliance simplification. ### Production Reality: What Should You Do If You Have Identifiable Data If you can get away with the prevention strategies above, that's ideal. But most production systems require storing identifiable data for core business functions. #### The GDPR Compliance Checklist 1. Latest Snapshot - Ensure all delete files are merged efficiently. 2. Old Snapshots - Ensure old snapshots are deleted, and branches, backups and tagged snapshots are taken care of. 3. Orphan Files - Ensure both orphan files and orphan delete files are cleaned up. #### Deleting Data from the Latest Snapshot ### Merge-on-Read (MOR) vs Copy-on-Write (COW) In Iceberg, there are two write strategies, that determine how Iceberg treats this data - it is either deleted immediately with COW or it leaves a delete file with MOR. We can utilize the right strategy to our advantage when thinking about how to setup the right compliance strategy, depending on our use-case. ![](https://cdn.prod.website-files.com/68ef977c5ec5483ef208e8b3/68efac2d7d5065b5e67634ab_688a598c990851229aa26ad0_Screenshot%25202025-07-30%2520at%25201.41.42%25E2%2580%25AFPM.avif) **Recommendation**: If you have smaller data volumes, infrequent deletes, and don't mind paying a premium for simplicity, go with COW. If you have large data volumes or frequent delete operations, use MOR with a robust compaction strategy that ensures delete files are compacted after a maximum of 30 days. > **Pro Tip**: You can setup your write strategy to MOR only for delete operations, by using ‘write.delete.mode: merge-on-read’ ### Bucketing for Performance Bucketing by user\_id can significantly improve GDPR deletion performance by clustering related user data together. Instead of scanning entire partitions to find a user's records, you only need to check specific buckets. **How it works:** ` `This creates 16 buckets based on user\_id hash, so all records for a given user in a given day end up in the same bucket within each time partition. **Why bucketing helps:** - **Faster deletions**: Target specific buckets instead of scanning entire partitions - **Lower costs**: Rewrite fewer files during deletion operations - **Predictable performance**: Deletion time becomes proportional to user activity, not table size **When to use bucketing:** - Tables where you frequently query or delete by user\_id - Large tables where full partition scans are expensive - When deletion performance is more important than general query performance (usually raw data) **When NOT to use bucketing:** - Tables with diverse query patterns that don't center on user\_id - Small tables where bucketing overhead outweighs benefits - When most queries aggregate across all users - Tables accessed primarily by time-based queries without user filtering **Trade-off**: Bucketing optimizes for user-centric operations but can slow down queries that don't filter by user\_id. ### Handling Historical Snapshots Historical snapshots are often overlooked but critical for GDPR compliance. Even after deleting user data from the current snapshot, it may still exist in old snapshots accessible via time travel queries. #### **Standard Approach:** For most use cases, setting up a standard snapshot retention policy that automatically cleans up old snapshots after 5 days would suffice. This handles the majority of GDPR scenarios by ensuring deleted data doesn't persist in historical snapshots beyond your retention window. #### **Special Cases: Custom Snapshot Strategies** If you have longer retention requirements for business reasons (backups, compliance audits, debugging), you need additional deletion processes: - **Branching**: Delete from all branches, not just main - **Long-term snapshots**: Run deletion commands against retained snapshots - **Backup policies**: Coordinate with your backup retention to ensure consistency The key is ensuring your snapshot retention aligns with GDPR requirements - deleted data shouldn't be accessible through any historical view. ### Orphan Files Personal data could also be found in orphan files, and cleaning orphan files is a critical step. It’s important to cleanup both Orphan Files, and Orphan Delete files. #### Monitoring and Verification Setting up proper monitoring is crucial to ensure tables remain GDPR compliant. Without visibility into your data lifecycle, you can't guarantee compliance or optimize your deletion strategy. #### Essential Metrics to Track **Oldest Snapshot Date** **** Monitor the age of your oldest snapshot per table to ensure retention policies are working: Set alerts when snapshots exceed your retention policy (e.g., >5 days for most tables). **Delete Files Count and Age** **** Track delete files to monitor MOR table health and compaction effectiveness: High delete file counts or old delete files indicate compaction issues that could violate GDPR compliance. **Rewrite Efficiency Tracking** **** Monitor data rewrite operations to optimize your deletion strategy: **`rewrite_efficieny = data_removed / data_rewritten # A number between 1 to 0. Higher is more efficient.** **`** Low rewrite efficiency (many files rewritten for few rows removed) suggests your partitioning or bucketing strategy may need adjustment. #### Recommended Alerting Thresholds - **Snapshots older than retention policy + 24 hours** - **Delete files older than compaction schedule + 24 hours** - **Rewrite efficiency below 0.1** - **Failed compaction jobs** (critical for MOR tables) Proper monitoring ensures you catch compliance issues before they become violations and helps optimize your deletion strategy over time. I hope this guide gives you a good understanding for how to create a GDPR compliant Iceberg lakehouse. If you have any suggestions, or questions, feel free to email me, or message me on [**LinkedIn**](https://www.linkedin.com/in/yossi-reitblat/). ## One Last Option - A Managed Iceberg Solution (Ryft) If you don’t want the responsibility of managing GDPR compliance yourself, don’t want to go through the hustle of setting up a GDPR compliance mechanism for each one of your tables, monitor it and constantly adjust based on the results - you can also use a solution like **Ryft**. [**Ryft**](https://ryft.io/) automatically ensures that your tables are GDPR compliant by: 1. Triggering efficient compaction jobs, to merge delete files at the right time. 2. Ensuring Snapshot Expiration & Orphan File Cleanup. 3. Ensuring the safe deletion of data from backups and branches. It’s as simple as a checkbox. ![](https://cdn.prod.website-files.com/68ef977c5ec5483ef208e8b3/68efac2d7d5065b5e67634b0_688a58baa3fc1b5974286d67_Screenshot%25202025-07-30%2520at%25201.38.29%25E2%2580%25AFPM.avif) Ryft Compliance Cleanup Interested in learning more? Feel free to [schedule a demo](https://ryft.io/demo) or email us at [hi@ryft.io](mailto:hi@ryft.io) Table of Contents [What GDPR Compliance Actually Means](https://www.ryft.io/blog/gdpr-compliance-with-apache-iceberg-a-practical-guide#what-gdpr-compliance-actually-means) [Why Is It Complicated with Iceberg?](https://www.ryft.io/blog/gdpr-compliance-with-apache-iceberg-a-practical-guide#why-is-it-complicated-with-iceberg) [1\. Multiple Data Copies via Snapshots](https://www.ryft.io/blog/gdpr-compliance-with-apache-iceberg-a-practical-guide#1-multiple-data-copies-via-snapshots) [2\. User Data is Scattered Across TBs of Files and Partitions](https://www.ryft.io/blog/gdpr-compliance-with-apache-iceberg-a-practical-guide#2-user-data-is-scattered-across-tbs-of-files-and-partitions) [3\. Copy-on-Write vs Merge-on-Read](https://www.ryft.io/blog/gdpr-compliance-with-apache-iceberg-a-practical-guide#3-copy-on-write-vs-merge-on-read) [Setting Up a GDPR-Compliant Lakehouse](https://www.ryft.io/blog/gdpr-compliance-with-apache-iceberg-a-practical-guide#setting-up-a-gdpr-compliant-lakehouse) [Avoid the Problem If You Can](https://www.ryft.io/blog/gdpr-compliance-with-apache-iceberg-a-practical-guide#avoid-the-problem-if-you-can) [Production Reality: What Should You Do If You Have Identifiable Data](https://www.ryft.io/blog/gdpr-compliance-with-apache-iceberg-a-practical-guide#production-reality-what-should-you-do-if-you-have-identifiable-data) [Merge-on-Read (MOR) vs Copy-on-Write (COW)](https://www.ryft.io/blog/gdpr-compliance-with-apache-iceberg-a-practical-guide#merge-on-read-mor-vs-copy-on-write-cow) [Bucketing for Performance](https://www.ryft.io/blog/gdpr-compliance-with-apache-iceberg-a-practical-guide#bucketing-for-performance) [Handling Historical Snapshots](https://www.ryft.io/blog/gdpr-compliance-with-apache-iceberg-a-practical-guide#handling-historical-snapshots) [Orphan Files](https://www.ryft.io/blog/gdpr-compliance-with-apache-iceberg-a-practical-guide#orphan-files) [One Last Option - A Managed Iceberg Solution (Ryft)](https://www.ryft.io/blog/gdpr-compliance-with-apache-iceberg-a-practical-guide#one-last-option-a-managed-iceberg-solution-ryft) Get the latest posts straight to your inbox Thank you! Your submission has been received! ## Browse other blogs ![Apache Iceberg V3: Is It Ready?](https://cdn.prod.website-files.com/68ef977c5ec5483ef208e8b3/69415e4b45acad26189a17d2_Blog%20Coverv3.png) Engineering ## Apache Iceberg V3: Is It Ready? Apache Iceberg V3 is a huge step forward for the lakehouse ecosystem. The V3 specification was finalized and ratified earlier this year, bringing several long-awaited capabilities into the core of the format: efficient row-level deletes, built-in row lineage, better handling of semi-structured data, and the beginnings of native encryption. This post breaks down the major features, the current state of implementation, and what this means for real adoption. ![](https://cdn.prod.website-files.com/68ef977c5ec5483ef208e8b3/68f1061b22ca6c79303ddfab_Auteur_pic_01%20(3).avif) Guy Yasoor December 16, 2025 December 17, 2025 [blog](https://www.ryft.io/blog/apache-iceberg-v3-is-it-ready) ![Streaming with Apache Iceberg: The Operational Problems at Scale](https://cdn.prod.website-files.com/68ef977c5ec5483ef208e8b3/692895683e8be6007b3e1902_Blog%20Cover.png) Engineering ## Streaming with Apache Iceberg: The Operational Problems at Scale Streaming into Iceberg creates three operational problems most teams don't see coming: small files pile up faster than you can compact them, storage costs climb because you're paying for data you've already replaced, and merges take much longer than they should. After seeing these problems in real production data lakes, we decided to share more about the causes and possible solutions. ![](https://cdn.prod.website-files.com/68ef977c5ec5483ef208e8b3/68f105fe51b1bd3a206f8f26_Auteur_pic_01%20(2).avif) Yuval Yogev December 8, 2025 December 1, 2025 [blog](https://www.ryft.io/blog/streaming-with-apache-iceberg-the-operational-problems-at-scale) ![How to Choose an Apache Iceberg Catalog](https://cdn.prod.website-files.com/68ef977c5ec5483ef208e8b3/692068eb2d13507cc7245b1e_Blog%20Cover.png) Engineering ## How to Choose an Apache Iceberg Catalog Apache Iceberg has become the table format of choice for building open data lakehouses. It solves long-standing problems around ACID transactions and engine interoperability. This post covers how to choose an Iceberg Catalog for production. ![](https://cdn.prod.website-files.com/68ef977c5ec5483ef208e8b3/68f1061b22ca6c79303ddfab_Auteur_pic_01%20(3).avif) Guy Yasoor December 8, 2025 November 21, 2025 [blog](https://www.ryft.io/blog/how-to-choose-an-apache-iceberg-catalog) ## See Ryft in Action The only solution that automatically maintains and optimizes your Iceberg tables based on usage, no lock-in, no manual tuning [See Ryft in Action](https://www.ryft.io/see-ryft-in-action) ![screening illustration](https://cdn.prod.website-files.com/68cf7feaa53a47caf3256f08/68ee7e458fe1bc5e963b6380_Frame%201707484444.avif) ![blog cta ](https://cdn.prod.website-files.com/68cf7feaa53a47caf3256f08/68efc958cdd00391780e03c2_Frame%201707484444%20(1).avif) [home](https://www.ryft.io/) ## Apache Iceberg Commit Conflicts [Ryft Blog](https://www.ryft.io/blog) Engineering # Handling Commit Conflicts in Apache Iceberg: Patterns and Fixes ![](https://cdn.prod.website-files.com/68ef977c5ec5483ef208e8b3/68f105da696b0f9017b6c9ed_Auteur_pic_01.avif) Yossi Reitblat October 23, 2025 ![](https://cdn.prod.website-files.com/68ef977c5ec5483ef208e8b3/68fa1827f28db910371a26d5_commit_conflict_cover.png) Commit conflicts in Apache Iceberg are one of those problems that seem rare - until you start operating at scale. The first time a long-running compaction job fails after hours of compute, or a CDC pipeline spends half its time retrying commits, you realize this isn’t a corner case. It’s a core operational challenge that directly impacts cost, latency, and reliability. This post covers what commit conflicts are, why they happen, and how to fix them without creating new problems in the process. ## Why Commit Conflicts Matter In Iceberg, multiple concurrent writers – streaming ingestion, compaction, CDC merges, and deletes – all operate independently. That independence is part of Iceberg’s design. But when several of these writers target the same table, conflicts scale fast. At small scale, retries handle the occasional overlap. At large scale, retries turn into endless loops: compaction jobs fail after hours, streaming pipelines fall behind, and delete operations never complete. These failures mean: - **Pipeline failures:** Streaming or CDC pipelines encountering repeated conflicts fail entirely, stopping data writes and breaking downstream dependencies. **** - **Wasted compute time:** a single failed compaction can burn thousands of cluster-hours. - **Higher query costs:** un-compacted tables accumulate small files that make queries slower and more expensive. - **Compliance risk:** failed deletes mean data that should be gone remains accessible. ## What Causes Commit Conflicts Iceberg uses **optimistic concurrency control**. Each writer assumes it’s the only one modifying the table and only checks for conflicts at commit time using an atomic compare-and-swap on the table’s metadata pointer. Here’s what happens: 1. Writer A starts processing with snapshot version 2. 2. Writer B finishes first and commits, moving the pointer to version 3. 3. Writer A tries to commit, expecting version 2 - but finds version 3. Iceberg rejects it with a CommitFailedException. ![](https://cdn.prod.website-files.com/68ef977c5ec5483ef208e8b3/68fa18390bb1e5cd2d95c36d_68f8ed9807b74891c2981170_Diagram.png) Iceberg Commit Conflict By design, this protects table integrity. But at scale, it also exposes two distinct types of conflicts. ### Metadata Conflicts Metadata-only conflicts happen when another writer updates the table metadata (for example, by adding new files or statistics) without touching the same data files. Iceberg can resolve these automatically by refreshing metadata and retrying the commit. Default retry settings - four attempts with exponential backoff from 100ms to 60s - work well for small to medium workloads. Once concurrency increases, those defaults stop being enough. ### Data Conflicts Data conflicts occur when two writers modify the same data files or partitions. These can’t be resolved automatically. The entire operation must restart. This is what makes commit conflicts painful: a compaction job or merge that runs for two hours can fail seconds before completion, wasting all that compute time. As concurrency grows, so does the ratio of unresolvable data conflicts. ## Common Conflict Patterns at Scale The three scenarios below account for 90% of production commit conflicts: ### Compaction During Active Writes A compaction job rewrites many small files into larger ones. If your streaming pipeline keeps writing to the same partition during that process, you’re guaranteed to collide. **Example:** You compact data for event\_date = '2025-10-10' for two hours, while your ingestion job commits 47 micro-batches to that same partition. When compaction tries to commit, it finds version N+47. Iceberg rejects it - data conflict. The result: two hours of compute, zero results. The small files remain, and the next compaction attempt is even slower. ### Deletes on Live Data Compliance or cleanup deletes can run for tens of minutes across large tables. If ingestion continues writing new data into those same partitions, the delete job fails. That means compliance-sensitive data stays in your system longer than it should - or you halt ingestion to guarantee deletes, losing fresh data in the process. ### CDC and MERGE Workloads CDC pipelines and merge operations are especially conflict-prone. They perform upserts across many partitions, often in parallel. Even small overlaps between concurrent merges trigger data conflicts. The result is high retry rates, longer latency, and significant wasted compute. ### The Compounding Cost Problem Once conflicts start happening, they create a feedback loop. Failed compactions leave small files behind. More small files make future compactions slower. Longer jobs mean larger conflict windows, leading to more failures. Query performance degrades, costs rise, and the cycle continues. A table that costs $10 per query when healthy can easily climb to hundreds per query when compaction falls behind. ## How to Fix Commit Conflicts The way to address commit conflicts depends on workload patterns, table layout, and how concurrency is managed. In production, this is done through careful job timing, partition design, and retry configuration. ### 1\. Enable Partial Progress for Long-Running Jobs For long-running jobs like compaction, the worst-case scenario is losing hours of work to a conflict near the end. Enabling partial progress changes that. ```sql CALL system.rewrite_data_files( table => '{table}' options => map('partial-progress.enabled', 'true', 'partial-progress.max-commits', 10) ) ``` With partial progress, Iceberg commits work in chunks. If a conflict occurs near the end, only the last chunk retries. **Consideration**: Each partial commit creates a snapshot. You’ll need to manage snapshot retention more actively: - Set `history.expire.min-snapshots-to-keep`and/or` history.expire.max-snapshot-age-ms`to control snapshots buildup Use this setting when the cost of retrying outweighs the cost of extra snapshots. ### 2\. Shrink the Conflict Window Sometimes the solution is just to accelerate the running job by doing less. Smaller file groups complete quicker, reducing the time a job stays vulnerable to conflicts. ```sql CALL system.rewrite_data_files( table => '{table}' options => map('max-file-group-size-bytes', 1073741824) ) ``` Also, tune retry behavior for high-concurrency environments: ```javascript commit.retry.num-retries=8 commit.retry.min-wait-ms=200 commit.retry.max-wait-ms=120000 ``` More retries and longer backoff periods help smooth out transient metadata conflicts without overloading the catalog. Again, this comes with a consideration: more frequent commits mean more snapshots. Balance accordingly. ### 3\. Avoid Compacting Active Partitions This single adjustment eliminates most conflicts in streaming environments. Don’t compact partitions currently receiving writes. ```javascript CALL system.rewrite_data_files( table => 'events', where => 'event_date < current_date()' ); ``` Your streaming job writes to today’s partition; your compaction job targets older partitions that are stable. For CDC workloads, apply the same idea. Track which partitions actually received updates and compact only those. Compacting everything, every time, just creates unnecessary conflict exposure. ## Detecting Conflict Symptoms You can usually spot conflict problems before they cascade. ### 1\. Excessive Retries Check your job logs for repeated CommitFailedException. ```javascript grep "CommitFailedException" /var/log/spark/* | wc -l ``` If you’re seeing dozens per day on a single table, your writers are fighting each other. ### 2\. Snapshot accumulation ```sql SELECT COUNT(*) FROM events.snapshots WHERE committed_at > current_date - INTERVAL '1' DAY; ``` More than 1000 snapshots per day on a single table might indicate partial progress misconfiguration. ### 3\. File count per partition ```sql SELECT partition, COUNT(*) AS file_count FROM events.files GROUP BY partition ORDER BY file_count DESC; ``` High file counts mean compaction is falling behind. Each query opens more files, scans more metadata, and costs more. ## Conclusion Commit conflicts are a natural part of Iceberg’s optimistic concurrency model. At small scale they’re easy to miss; at large scale they’re the main reason compaction, deletes, and merges fail. Managing them means keeping conflict windows short, jobs predictable, and partition boundaries stable. With this in place, concurrency stops being a problem and becomes just another part of running Iceberg in production. Table of Contents [Why Commit Conflicts Matter](https://www.ryft.io/blog/handling-commit-conflicts-in-apache-iceberg-patterns-and-fixes#why-commit-conflicts-matter) [What Causes Commit Conflicts](https://www.ryft.io/blog/handling-commit-conflicts-in-apache-iceberg-patterns-and-fixes#what-causes-commit-conflicts) [Metadata Conflicts](https://www.ryft.io/blog/handling-commit-conflicts-in-apache-iceberg-patterns-and-fixes#metadata-conflicts) [Data Conflicts](https://www.ryft.io/blog/handling-commit-conflicts-in-apache-iceberg-patterns-and-fixes#data-conflicts) [Common Conflict Patterns at Scale](https://www.ryft.io/blog/handling-commit-conflicts-in-apache-iceberg-patterns-and-fixes#common-conflict-patterns-at-scale) [Compaction During Active Writes](https://www.ryft.io/blog/handling-commit-conflicts-in-apache-iceberg-patterns-and-fixes#compaction-during-active-writes) [Deletes on Live Data](https://www.ryft.io/blog/handling-commit-conflicts-in-apache-iceberg-patterns-and-fixes#deletes-on-live-data) [CDC and MERGE Workloads](https://www.ryft.io/blog/handling-commit-conflicts-in-apache-iceberg-patterns-and-fixes#cdc-and-merge-workloads) [The Compounding Cost Problem](https://www.ryft.io/blog/handling-commit-conflicts-in-apache-iceberg-patterns-and-fixes#the-compounding-cost-problem) [How to Fix Commit Conflicts](https://www.ryft.io/blog/handling-commit-conflicts-in-apache-iceberg-patterns-and-fixes#how-to-fix-commit-conflicts) [1\. Enable Partial Progress for Long-Running Jobs](https://www.ryft.io/blog/handling-commit-conflicts-in-apache-iceberg-patterns-and-fixes#1-enable-partial-progress-for-long-running-jobs) [2\. Shrink the Conflict Window](https://www.ryft.io/blog/handling-commit-conflicts-in-apache-iceberg-patterns-and-fixes#2-shrink-the-conflict-window) [3\. Avoid Compacting Active Partitions](https://www.ryft.io/blog/handling-commit-conflicts-in-apache-iceberg-patterns-and-fixes#3-avoid-compacting-active-partitions) [Detecting Conflict Symptoms](https://www.ryft.io/blog/handling-commit-conflicts-in-apache-iceberg-patterns-and-fixes#detecting-conflict-symptoms) [1\. Excessive Retries](https://www.ryft.io/blog/handling-commit-conflicts-in-apache-iceberg-patterns-and-fixes#1-excessive-retries) [2\. Snapshot accumulation](https://www.ryft.io/blog/handling-commit-conflicts-in-apache-iceberg-patterns-and-fixes#2-snapshot-accumulation) [3\. File count per partition](https://www.ryft.io/blog/handling-commit-conflicts-in-apache-iceberg-patterns-and-fixes#3-file-count-per-partition) [Conclusion](https://www.ryft.io/blog/handling-commit-conflicts-in-apache-iceberg-patterns-and-fixes#conclusion) Get the latest posts straight to your inbox Thank you! Your submission has been received! ## Browse other blogs ![Apache Iceberg V3: Is It Ready?](https://cdn.prod.website-files.com/68ef977c5ec5483ef208e8b3/69415e4b45acad26189a17d2_Blog%20Coverv3.png) Engineering ## Apache Iceberg V3: Is It Ready? Apache Iceberg V3 is a huge step forward for the lakehouse ecosystem. The V3 specification was finalized and ratified earlier this year, bringing several long-awaited capabilities into the core of the format: efficient row-level deletes, built-in row lineage, better handling of semi-structured data, and the beginnings of native encryption. This post breaks down the major features, the current state of implementation, and what this means for real adoption. ![](https://cdn.prod.website-files.com/68ef977c5ec5483ef208e8b3/68f1061b22ca6c79303ddfab_Auteur_pic_01%20(3).avif) Guy Yasoor December 16, 2025 December 17, 2025 [blog](https://www.ryft.io/blog/apache-iceberg-v3-is-it-ready) ![Streaming with Apache Iceberg: The Operational Problems at Scale](https://cdn.prod.website-files.com/68ef977c5ec5483ef208e8b3/692895683e8be6007b3e1902_Blog%20Cover.png) Engineering ## Streaming with Apache Iceberg: The Operational Problems at Scale Streaming into Iceberg creates three operational problems most teams don't see coming: small files pile up faster than you can compact them, storage costs climb because you're paying for data you've already replaced, and merges take much longer than they should. After seeing these problems in real production data lakes, we decided to share more about the causes and possible solutions. ![](https://cdn.prod.website-files.com/68ef977c5ec5483ef208e8b3/68f105fe51b1bd3a206f8f26_Auteur_pic_01%20(2).avif) Yuval Yogev December 8, 2025 December 1, 2025 [blog](https://www.ryft.io/blog/streaming-with-apache-iceberg-the-operational-problems-at-scale) ![How to Choose an Apache Iceberg Catalog](https://cdn.prod.website-files.com/68ef977c5ec5483ef208e8b3/692068eb2d13507cc7245b1e_Blog%20Cover.png) Engineering ## How to Choose an Apache Iceberg Catalog Apache Iceberg has become the table format of choice for building open data lakehouses. It solves long-standing problems around ACID transactions and engine interoperability. This post covers how to choose an Iceberg Catalog for production. ![](https://cdn.prod.website-files.com/68ef977c5ec5483ef208e8b3/68f1061b22ca6c79303ddfab_Auteur_pic_01%20(3).avif) Guy Yasoor December 8, 2025 November 21, 2025 [blog](https://www.ryft.io/blog/how-to-choose-an-apache-iceberg-catalog) ## See Ryft in Action The only solution that automatically maintains and optimizes your Iceberg tables based on usage, no lock-in, no manual tuning [See Ryft in Action](https://www.ryft.io/see-ryft-in-action) ![screening illustration](https://cdn.prod.website-files.com/68cf7feaa53a47caf3256f08/68ee7e458fe1bc5e963b6380_Frame%201707484444.avif) ![blog cta ](https://cdn.prod.website-files.com/68cf7feaa53a47caf3256f08/68efc958cdd00391780e03c2_Frame%201707484444%20(1).avif) [home](https://www.ryft.io/) ## Data Retention Compliance [Ryft Blog](https://www.ryft.io/blog) News # Announcing Ryft Data Retention & Compliance Enforcement for Apache Iceberg ![](https://cdn.prod.website-files.com/68ef977c5ec5483ef208e8b3/68f105fe51b1bd3a206f8f26_Auteur_pic_01%20(2).avif) Yuval Yogev December 8, 2025 ![](https://cdn.prod.website-files.com/68ef977c5ec5483ef208e8b3/6936da25feec7bdfd478e008_Blog%20Cover.png) Today, we’re introducing two new capabilities in Ryft: **Automated Data Retention** and **Data Compliance Enforcement** for Apache Iceberg™. These features integrate directly into the Ryft platform to ensure efficient, policy-driven data deletion and compliance, working seamlessly alongside table maintenance and optimization. ## Deleting Data in Iceberg Is Much Harder Than It Looks Deleting data in Apache Iceberg is deceptively complex. Teams usually need to delete data for two reasons: **1\. Cost & operational efficiency** \- drop old data to keep storage and compute costs predictable **** **2\. Compliance** \- ensure regulated data is entire _ly removed_ (GDPR, CCPA, internal policies) Both sound straightforward: “delete rows older than X” or “delete this user’s data.” But with Iceberg, deletion needs to be done carefully and efficiently. Doing it wrong can corrupt tables, leave non-compliant data behind, or waste massive amounts of compute. Customers required to comply with data regulations had to build custom workflows to locate soft deletes, consolidate them, rewrite affected partitions, and verify that old snapshots no longer reference the data. **Data Retention** defines how long data should live: 7 days, 30 days, 13 months, etc. Although object storage systems support deletion policies, you [can’t just use them directly on Iceberg tables](https://www.ryft.io/blog/data-retention-in-apache-iceberg-implementation-details-and-best-practices), as the tables [**will get corrupted**](https://www.ryft.io/blog/how-to-fix-corrupted-iceberg-tables), because Iceberg metadata still references the files. **Data Compliance** requirements like GDPR and CCPA demand **physical deletion**, not soft deletion. With data stored across thousands of files and referenced from older snapshots and backups, [making sure Iceberg tables are compliant is an operational challenge](https://www.ryft.io/blog/gdpr-compliance-with-apache-iceberg-a-practical-guide). ## How The Ryft Platform Solves It ### 1\. Automated Data Retention **Ryft turns Iceberg retention into a fully automated, scalable operation**. Instead of building custom pipelines or manually coordinating rewrites, you define the policy, and Ryft handles everything else. Retention policies can be applied at any scope: a single table, a group of tables, or your entire lake.Ryft continuously evaluates tables and automatically identifies which partitions contain data past the retention window. Data retention is performed in a partition-aware way, that is highly efficient as they do not require excessive data scans. Automated Data Retention is also safe by default - dry runs and policy validation are performed to prevent accidental misconfiguration that can lead to data loss. Iceberg metadata is used to verify the expected amount of data to be deleted relative to the actual data in the table and makes sure there is no anomaly. ![](https://cdn.prod.website-files.com/68ef977c5ec5483ef208e8b3/6936d9b3a272acd7cec21420_image%20(12).png) Configuring Data Retention with Ryft ### 2\. Compliance Enforcement **Ryft compliance cleanup consolidates all the complexity of keeping Iceberg tables compliant into a single checkbox.** You mark a table as “compliance-enforced,” and Ryft automatically ensures that any deletion is physically removed from the table: no special pipelines, no coordination with ingestion, no manual rewrites. This works consistently even across **thousands of tables**, with different schemas, workloads, and ingestion patterns, both for live and historical data. ![](https://cdn.prod.website-files.com/68ef977c5ec5483ef208e8b3/6936d9ceab2c1650f5717012_image%20(13).png) Configuring Compliance Cleanup with Ryft ### 3\. Unified With the Rest of Ryft Retention and compliance enforcement aren't standalone features - they're integrated into Ryft's intelligent optimization engine. Everything runs in coordination: - **Adaptive Optimization** - **Retention policies** - **Compliance cleanup** - **Snapshot expiration** - **Metadata cleanup** This means no conflicts or duplicate work, and no manual scheduling to prevent operations from stepping on each other. ## **From Operational Burden to Autonomous Infrastructure** Before Ryft, teams managing data deletion faced a choice: - **Build complex systems** (months of engineering, ongoing maintenance) - **Accept non-compliance** (regulatory risk, audit failures) Ryft eliminates that choice. Automated retention and compliance enforcement handle what used to require dedicated engineering resources - and they scale to thousands of tables without additional overhead. Our customers have already been running these in production with multi-petabyte lakes. [Book a call with us](https://www.ryft.io/see-ryft-in-action) to see it in action **.** These capabilities are available to all Ryft customers today. Whether you're managing compliance requirements or optimizing costs, we'll show you how to implement policies that deliver immediate results. Table of Contents [Deleting Data in Iceberg Is Much Harder Than It Looks](https://www.ryft.io/blog/announcing-ryft-data-retention-compliance-enforcement-for-apache-iceberg#deleting-data-in-iceberg-is-much-harder-than-it-looks) [How The Ryft Platform Solves It](https://www.ryft.io/blog/announcing-ryft-data-retention-compliance-enforcement-for-apache-iceberg#how-the-ryft-platform-solves-it) [1\. Automated Data Retention](https://www.ryft.io/blog/announcing-ryft-data-retention-compliance-enforcement-for-apache-iceberg#1-automated-data-retention) [2\. Compliance Enforcement](https://www.ryft.io/blog/announcing-ryft-data-retention-compliance-enforcement-for-apache-iceberg#2-compliance-enforcement) [3\. Unified With the Rest of Ryft](https://www.ryft.io/blog/announcing-ryft-data-retention-compliance-enforcement-for-apache-iceberg#3-unified-with-the-rest-of-ryft) [From Operational Burden to Autonomous Infrastructure](https://www.ryft.io/blog/announcing-ryft-data-retention-compliance-enforcement-for-apache-iceberg#from-operational-burden-to-autonomous-infrastructure) Get the latest posts straight to your inbox Thank you! Your submission has been received! ## Browse other blogs ![Announcing Ryft Data Retention & Compliance Enforcement for Apache Iceberg](https://cdn.prod.website-files.com/68ef977c5ec5483ef208e8b3/6936da25feec7bdfd478e008_Blog%20Cover.png) News ## Announcing Ryft Data Retention & Compliance Enforcement for Apache Iceberg Today, we’re introducing two new capabilities in Ryft: Automated Data Retention and Data Compliance Enforcement for Apache Iceberg™. These features integrate directly into the Ryft platform to ensure efficient, policy-driven data deletion and compliance, working seamlessly alongside table maintenance and optimization. ![](https://cdn.prod.website-files.com/68ef977c5ec5483ef208e8b3/68f105fe51b1bd3a206f8f26_Auteur_pic_01%20(2).avif) Yuval Yogev December 8, 2025 December 8, 2025 [blog](https://www.ryft.io/blog/announcing-ryft-data-retention-compliance-enforcement-for-apache-iceberg) ![Announcing Ryft Adaptive Optimization](https://cdn.prod.website-files.com/68ef977c5ec5483ef208e8b3/68efaaa4554eca8245e5f2f2_68caa4aa095abc61c9362c0b_Blog%2520Cover%2520(1).avif) News ## Announcing Ryft Adaptive Optimization Today, we’re officially introducing Ryft Adaptive Optimization - always-on, dynamic optimization engine for Apache Iceberg™. Our engine continuously compacts, rewrites, indexes, and reorders data based on how your tables are actually used, delivering up to 5× faster queries, 10x storage reduction, and 7x better compaction efficiency compared to other engines. ![](https://cdn.prod.website-files.com/68ef977c5ec5483ef208e8b3/68f105da696b0f9017b6c9ed_Auteur_pic_01.avif) Yossi Reitblat October 23, 2025 September 17, 2025 [blog](https://www.ryft.io/blog/announcing-ryft-adaptive-optimization) ![Unlocking Iceberg management for everyone](https://cdn.prod.website-files.com/68ef977c5ec5483ef208e8b3/68efac34c4925bbf47b6a113_689363b8bb6dd97ccd38b882_686d09873da2afe9164ed0ba_cover.avif) News ## Unlocking Iceberg management for everyone Ryft, in many ways, is a story 15 years in the making. Yuval Yogev, Guy Gadon and I went to the same high school, worked together at 8200, building high-scale data infrastructure, and went our separate ways - all to realize that we really enjoy solving complicated data infrastructure problems together with the people we love. ![](https://cdn.prod.website-files.com/68ef977c5ec5483ef208e8b3/68f105da696b0f9017b6c9ed_Auteur_pic_01.avif) Yossi Reitblat October 27, 2025 July 8, 2025 [blog](https://www.ryft.io/blog/unlocking-iceberg-management-for-everyone) ## See Ryft in Action The only solution that automatically maintains and optimizes your Iceberg tables based on usage, no lock-in, no manual tuning [See Ryft in Action](https://www.ryft.io/see-ryft-in-action) ![screening illustration](https://cdn.prod.website-files.com/68cf7feaa53a47caf3256f08/68ee7e458fe1bc5e963b6380_Frame%201707484444.avif) ![blog cta ](https://cdn.prod.website-files.com/68cf7feaa53a47caf3256f08/68efc958cdd00391780e03c2_Frame%201707484444%20(1).avif) [home](https://www.ryft.io/) ## Data Retention in Iceberg [Ryft Blog](https://www.ryft.io/blog) Engineering # Data Retention in Apache Iceberg: Implementation Details and Best Practices ![](https://cdn.prod.website-files.com/68ef977c5ec5483ef208e8b3/68f105fe51b1bd3a206f8f26_Auteur_pic_01%20(2).avif) Yuval Yogev October 3, 2025 ![](https://cdn.prod.website-files.com/68ef977c5ec5483ef208e8b3/68efaaa43a9a4f90a3cc617a_68e01bb701e38e364ba20180_retention_cover.avif) Data retention in Apache Iceberg is one of those critical operations that seems simple until you implement it at scale. Delete old data, save money, stay compliant - straightforward enough. But the implementation details matter, and getting them wrong can mean failed compliance audits, runaway storage costs, or accidentally purging the wrong data. This post covers the practical details: when retention matters, how to implement it correctly, and the common pitfalls that catch teams off guard. ## Why Perform Data Retention Data retention defines how long data should be kept before it's automatically deleted. Common motivations include: - **Compliance:** GDPR, CCPA, and other regulations require data deletion after defined periods. Retention policies ensure you're not holding data beyond its legal lifetime - **Storage optimization:** Retaining only recent data cuts storage costs by 30-80%, especially for append-heavy workloads like logs or telemetry - **Compute guardrails:** Limiting data volume keeps queries and maintenance operations predictable. A forgotten filter on a 5-year table versus a 90-day table is the difference between a $10 and $1,000 query ## Object Storage - Based Approaches Data retention is not a new concept - cloud object stores like Amazon S3, Google Cloud Storage, and Azure Blob have supported **lifecycle policies** for years. These allow you to automatically delete objects after a given period (for example, delete objects older than one year). However, with **open table formats like Apache Iceberg**, these storage-level policies are not sufficient, and can actually corrupt your tables. There are two main reasons for that: 1. **Metadata integrity:** In Iceberg, each data file is referenced by one or more manifest files, which in turn are tracked by snapshots. If an object-store lifecycle policy deletes a data file that’s still referenced in the table’s metadata, the table becomes inconsistent or unreadable. (More on that on our series about [data corruption](https://www.ryft.io/blog/iceberg-table-corruption-and-data-loss-in-the-wild-part-1)) 2. **Logical vs. physical time:** Retention in analytical systems should be based on **application-level timestamps** (e.g., `event_time`, `ingest_date`), not on the object’s creation or modification time in storage. Object creation time says nothing about the semantic age of the data, and therefore can’t be used safely for regulatory or analytical retention policies. For these reasons, data retention in Iceberg must be performed through the table layer using Iceberg’s metadata and expressions, rather than directly at the storage layer. ## How to Implement Retention in Iceberg ### Identify Retention Requirements Not every table needs retention. Focus on: - Fact or event tables with timestamp columns - Audit and operational logs - Intermediate datasets in analytics pipelines - Tables with compliance requirements For each table, define the retention period based on business needs: - 30 days for raw telemetry - 180 days for processed analytics - 365 days for compliance-archived events ### Execute Time-Based Deletes Iceberg supports row-level deletes using SQL or API-based expressions: ``This marks all rows older than 90 days for deletion. The actual implementation varies significantly based on your table's partitioning scheme and write mode, which we'll explore in the following sections. ## Partition-Based vs Row-Based Retention The performance of retention operations depends fundamentally on whether your retention column aligns with your partition column. ### Partition-Based Retention When your retention column is also your partition column, Iceberg performs highly efficient metadata-only operations: This operation: - Only modifies metadata pointers - Completes in seconds, regardless of data volume - Doesn't rewrite any data files - Scales to petabytes without performance degradation Iceberg simply removes references to the partitions from the table metadata. The data files remain in storage until snapshot expiration runs, but they're immediately invisible to queries. ### Row-Based Retention When retention targets a non-partition column, Iceberg must inspect and potentially rewrite data files: This operation: - Scans all partitions to find matching rows - Creates delete files or rewrites data files (depending on table configuration) - Performance degrades linearly with table size - Can take hours on large tables ### The Performance Impact The difference in compute time is dramatic: - **Partition-based deletion** on a 10TB table: ~5 seconds - **Row-based deletion** on the same table: 2-4 hours This 100-1000x difference in execution time is why partition design matters. If you know you'll need time-based retention, partition by time. **Important decision point:** If you're implementing row-based retention on large tables, consider repartitioning by your retention column first. The one-time migration cost often pays for itself within months. **Note on metadata-only deletes:** These only work when your filter predicate can be fully resolved at the partition level. For example, if your table is partitioned by `event_date` (day granularity), the query `WHERE event_date < '2024-01-01'` can use metadata-only deletion because Iceberg knows exactly which partition files to drop. However, if you use a timestamp column that's not the partition key, like `WHERE event_timestamp < '2024-01-01 00:00:00'`, Iceberg must scan the data files to find matching rows, triggering the slower row-based deletion path. ## Copy-on-Write vs Merge-on-Read Implications Your table's write mode affects how deletes are physically handled: ### Copy-on-Write (CoW) Tables - Rewrites affected data files immediately during the delete operation - Storage is reclaimed after snapshot expiration ### Merge-on-Read (MoR) Tables - Records deletes in separate delete files - Applies deletions at read time - Requires compaction to actually remove deleted data **Why this matters for retention:** With MoR tables, deleted data remains in your object store until compaction runs. If you're deleting for compliance, your compaction schedule must align with regulatory requirements. A 30-day GDPR requirement means nothing if compaction runs quarterly. **Important caveat:** Even with CoW configuration, engines like Trino and Athena default to creating delete files for performance reasons. ## Snapshot Expiration and File Purging Deletion in Iceberg is a two-phase process: 1. **Logical deletion**: Data becomes invisible to queries (immediate) 2. **Physical deletion**: Files are removed from storage (deferred) Physical deletion happens during snapshot expiration. If you retain snapshots for 7 days (for time-travel), deleted data remains in storage for at least 7 days. **Coordination requirement:** Your snapshot expiration schedule must align with compliance requirements. For 30-day GDPR compliance, you might need: - Daily / weekly retention deletes - 7-day snapshot retention - Weekly orphan file cleanup This ensures data is fully purged within the compliance window. ## Time Zone and Data Type Pitfalls ### Timezone Misalignment **The problem:** Different compute engines use different default timezones. Spark might use UTC while Trino uses your session timezone. **The solution:** Always be explicit about timezones in your retention queries:` ` ## Safety Measures and Backup Strategies Data retention is destructive by design. Here are two critical safety measures: ### Dry Run Testing Before implementing retention on production tables, always validate your logic:` ` ### Backup Coordination Backups are another safety net against wrong deletions. We'll cover Iceberg backup strategies in depth in an upcoming post, including incremental backup techniques, cross-region replication, and recovery procedures. ## Retention at Scale with Ryft Ryft automates data retention across your entire Iceberg environment. The platform help you identify which tables need retention, executes the optimal deletion strategy (partition-based when possible, row-based when necessary), and ensures compliance requirements are met. Every action is tracked and auditable, eliminating the operational complexity of managing retention at scale. This ensures that your lake is always compliant and cost-effective, with zero manual overhead. Want to see how it works? Talk to us about [Ryft's Intelligent Iceberg Management Platform](https://ryft.io/demo) Table of Contents [Why Perform Data Retention](https://www.ryft.io/blog/data-retention-in-apache-iceberg-implementation-details-and-best-practices#why-perform-data-retention) [Object Storage - Based Approaches](https://www.ryft.io/blog/data-retention-in-apache-iceberg-implementation-details-and-best-practices#object-storage-based-approaches) [How to Implement Retention in Iceberg](https://www.ryft.io/blog/data-retention-in-apache-iceberg-implementation-details-and-best-practices#how-to-implement-retention-in-iceberg) [Identify Retention Requirements](https://www.ryft.io/blog/data-retention-in-apache-iceberg-implementation-details-and-best-practices#identify-retention-requirements) [Execute Time-Based Deletes](https://www.ryft.io/blog/data-retention-in-apache-iceberg-implementation-details-and-best-practices#execute-time-based-deletes) [Partition-Based vs Row-Based Retention](https://www.ryft.io/blog/data-retention-in-apache-iceberg-implementation-details-and-best-practices#partition-based-vs-row-based-retention) [Partition-Based Retention](https://www.ryft.io/blog/data-retention-in-apache-iceberg-implementation-details-and-best-practices#partition-based-retention) [Row-Based Retention](https://www.ryft.io/blog/data-retention-in-apache-iceberg-implementation-details-and-best-practices#row-based-retention) [The Performance Impact](https://www.ryft.io/blog/data-retention-in-apache-iceberg-implementation-details-and-best-practices#the-performance-impact) [Copy-on-Write vs Merge-on-Read Implications](https://www.ryft.io/blog/data-retention-in-apache-iceberg-implementation-details-and-best-practices#copy-on-write-vs-merge-on-read-implications) [Copy-on-Write (CoW) Tables](https://www.ryft.io/blog/data-retention-in-apache-iceberg-implementation-details-and-best-practices#copy-on-write-cow-tables) [Merge-on-Read (MoR) Tables](https://www.ryft.io/blog/data-retention-in-apache-iceberg-implementation-details-and-best-practices#merge-on-read-mor-tables) [Snapshot Expiration and File Purging](https://www.ryft.io/blog/data-retention-in-apache-iceberg-implementation-details-and-best-practices#snapshot-expiration-and-file-purging) [Time Zone and Data Type Pitfalls](https://www.ryft.io/blog/data-retention-in-apache-iceberg-implementation-details-and-best-practices#time-zone-and-data-type-pitfalls) [Timezone Misalignment](https://www.ryft.io/blog/data-retention-in-apache-iceberg-implementation-details-and-best-practices#timezone-misalignment) [Safety Measures and Backup Strategies](https://www.ryft.io/blog/data-retention-in-apache-iceberg-implementation-details-and-best-practices#safety-measures-and-backup-strategies) [Dry Run Testing](https://www.ryft.io/blog/data-retention-in-apache-iceberg-implementation-details-and-best-practices#dry-run-testing) [Backup Coordination](https://www.ryft.io/blog/data-retention-in-apache-iceberg-implementation-details-and-best-practices#backup-coordination) [Retention at Scale with Ryft](https://www.ryft.io/blog/data-retention-in-apache-iceberg-implementation-details-and-best-practices#retention-at-scale-with-ryft) Get the latest posts straight to your inbox Thank you! Your submission has been received! ## Browse other blogs ![Apache Iceberg V3: Is It Ready?](https://cdn.prod.website-files.com/68ef977c5ec5483ef208e8b3/69415e4b45acad26189a17d2_Blog%20Coverv3.png) Engineering ## Apache Iceberg V3: Is It Ready? Apache Iceberg V3 is a huge step forward for the lakehouse ecosystem. The V3 specification was finalized and ratified earlier this year, bringing several long-awaited capabilities into the core of the format: efficient row-level deletes, built-in row lineage, better handling of semi-structured data, and the beginnings of native encryption. This post breaks down the major features, the current state of implementation, and what this means for real adoption. ![](https://cdn.prod.website-files.com/68ef977c5ec5483ef208e8b3/68f1061b22ca6c79303ddfab_Auteur_pic_01%20(3).avif) Guy Yasoor December 16, 2025 December 17, 2025 [blog](https://www.ryft.io/blog/apache-iceberg-v3-is-it-ready) ![Streaming with Apache Iceberg: The Operational Problems at Scale](https://cdn.prod.website-files.com/68ef977c5ec5483ef208e8b3/692895683e8be6007b3e1902_Blog%20Cover.png) Engineering ## Streaming with Apache Iceberg: The Operational Problems at Scale Streaming into Iceberg creates three operational problems most teams don't see coming: small files pile up faster than you can compact them, storage costs climb because you're paying for data you've already replaced, and merges take much longer than they should. After seeing these problems in real production data lakes, we decided to share more about the causes and possible solutions. ![](https://cdn.prod.website-files.com/68ef977c5ec5483ef208e8b3/68f105fe51b1bd3a206f8f26_Auteur_pic_01%20(2).avif) Yuval Yogev December 8, 2025 December 1, 2025 [blog](https://www.ryft.io/blog/streaming-with-apache-iceberg-the-operational-problems-at-scale) ![How to Choose an Apache Iceberg Catalog](https://cdn.prod.website-files.com/68ef977c5ec5483ef208e8b3/692068eb2d13507cc7245b1e_Blog%20Cover.png) Engineering ## How to Choose an Apache Iceberg Catalog Apache Iceberg has become the table format of choice for building open data lakehouses. It solves long-standing problems around ACID transactions and engine interoperability. This post covers how to choose an Iceberg Catalog for production. ![](https://cdn.prod.website-files.com/68ef977c5ec5483ef208e8b3/68f1061b22ca6c79303ddfab_Auteur_pic_01%20(3).avif) Guy Yasoor December 8, 2025 November 21, 2025 [blog](https://www.ryft.io/blog/how-to-choose-an-apache-iceberg-catalog) ## See Ryft in Action The only solution that automatically maintains and optimizes your Iceberg tables based on usage, no lock-in, no manual tuning [See Ryft in Action](https://www.ryft.io/see-ryft-in-action) ![screening illustration](https://cdn.prod.website-files.com/68cf7feaa53a47caf3256f08/68ee7e458fe1bc5e963b6380_Frame%201707484444.avif) ![blog cta ](https://cdn.prod.website-files.com/68cf7feaa53a47caf3256f08/68efc958cdd00391780e03c2_Frame%201707484444%20(1).avif) [home](https://www.ryft.io/) ## Iceberg Table Issues [Ryft Blog](https://www.ryft.io/blog) Engineering # Iceberg Table Corruption and Data Loss in the Wild: Part 1 ![](https://cdn.prod.website-files.com/68ef977c5ec5483ef208e8b3/68f105eb118c0310c1754068_Auteur_pic_01%20(1).avif) Omer Hadari July 16, 2025 ![](https://cdn.prod.website-files.com/68ef977c5ec5483ef208e8b3/68efac2d7195a84051d2f0fa_68779b8f2dbafc67dd6d41b4_New%2520blog%2520cover.avif) You’re a data engineer, and for a moment - everything looks calm. Dashboards are green, ingestion is steady, lags are low. Then, a worrying Slack message from a data scientist: “Do you know what this error means?” ![](https://cdn.prod.website-files.com/68ef977c5ec5483ef208e8b3/68efac2d7195a84051d2f08d_68ca9fd5f92eb4e4dfbc7fae_68779bc1707df6c3af376bdd_screenshot.avif) In this post, we want to share a story about a sneaky bug we encountered that caused **table corruption**, as well as **silent data loss** in Iceberg tables. If you're using Iceberg, if your ingestion is based on a streaming pipeline, if you're an AWS EMR user, or if you just like a good bug hunt - read on. ## Background Apache Iceberg is a specification for defining tables on Object Storage. It defines a set of **metadata files** which together describe a table. things such as its schema, how it's partitioned, and most importantly: **where its data resides.** Iceberg table data is stored in **data files**, which are typically Parquet files stored in Object Storage, such as S3. Engines (such as Spark, Flink, Trino and more) interact with Iceberg tables. They read and write metadata files to locate and manipulate data files when executing user queries. Users and applications access Iceberg tables through these engines. When issues occur with either data files or metadata files, it's the engine that reports the failure. ## The Problem Two different exceptions began appearing simultaneously. We suspected they were connected rather than just coincidental unrelated errors (as you'll see later - this assumption was right). Different queries triggered one or more of these exceptions: - **`java.io.IOException: java.lang.RuntimeException: s3://xxxxx.parquet is not a Parquet file. expected magic number at tail [80, 65, 82, 49] but found [x, x, x, x]`** Parquet is the most prominent format for data files in Iceberg (perhaps in data lakes generally). Parquet files are marked by a magic "PAR1" signature at both the beginning and end of the file. The first exception indicated that this expected magic signature wasn't found where it should have been, causing the read to fail. - **`software.amazon.awssdk.services.s3.model.S3Exception: The requested range is not satisfiable`** The second exception occurred when the reader requested an invalid portion of the S3 file. Interestingly, these exceptions weren't thrown at the same location in the code. We'll soon discover why. ## Data Loss and Table Corruption Ryft monitors Iceberg tables in real time to make smart decisions about what can be optimized and when. This monitoring revealed something concerning. Data files were being written to the same path multiple times with different sizes and record counts. In other words, **this was a data corruption issue**. Moreover, it was also a **silent data loss issue.** Files were overwritten during normal ingestion, without any explicit indication. When an exception was eventually thrown, it was when some of your data was already long gone. Thanks to our continuous monitoring, we could definitively confirm that files were being rewritten, and we could identify exactly when it happened and which files were affected. This would be impossible to determine by examining only the current state of the table. ## Reading Iceberg Code Files were being overwritten, and it was time to find out why. We compared the tables with this corruption issue to other tables which were fine. We noticed something interesting in the keys of the data files of the different tables: Here is an example key of an overwritten file: _`00011-12345-afafafaf-1010-1111-ddee-1e2e3e4e5e6e-00001.parquet`_ And here is an example of a key from a healthy table: _`00011-56321-efde3e12-1010-2356-efde-1e2e3e4e5e6e-`_ **_`274171`_** _`-00001.parquet`_ Note the part in bold. it looks like an extra portion of the generated ID, which is only present in keys from the healthy table. What's more, we saw files in the healthy table that had the exact same path **except** for the part in bold. To understand exactly how object keys are generated, we examined the source code. Searching the apache/iceberg repository, we quickly found the following [**pull request**](https://github.com/apache/iceberg/pull/9255/files): ![](https://cdn.prod.website-files.com/68ef977c5ec5483ef208e8b3/68efac2d7195a84051d2f090_68ca9fd5f92eb4e4dfbc7fab_68779b09d3b19f917acaad4b_screenshot1.avif) This confirmed our suspicion: it was a known issue. Because the problem did not cause errors during writes, and manifested as various different exceptions, it was particularly difficult to identify through online searches. The fix was introduced in Iceberg version 1.5.0. ### Explaining the Fix Before the pull request, file names were generated by using IDs that are unique **per Spark Application** alongside a running, in-memory counter\*\*.\*\* In batch mode, this isn't a problem. application IDs are unique, and in case of failure, the next run will have a different one. However, in Spark Streaming, application IDs persist even across restarts. When a streaming application restarted, the ID remained the same while the counter reset, causing name collisions and file overwrites. The pull request fixed the issue by adding the **epoch id**, which changes between restarts, to the name of generated files. ## Bonus Section: Explaining Exceptions This section goes deeper into the explanation for each of the three exceptions we encountered. It includes some elaboration on Iceberg metadata and how query engines use it to scan efficiently. _`If you're not interested in these technical details and just want to see how to protect yourself, feel free to skip directly to the conclusions section.`_ At the beginning of this post, we mentioned 2 different exceptions: - **`java.io.IOException: java.lang.RuntimeException: hdfs://xxxxx.parquet is not a Parquet file. expected magic number at tail [80, 65, 82, 49] but found [x, x, x, x]`** - **`software.amazon.awssdk.services.s3.model.S3Exception: The requested range is not satisfiable`** Knowing that files are being overwritten, all three exceptions made sense. Before we can explain why, we need to get a little bit into how query engines read Parquet files. ### Reading Parquets Remember we mentioned Iceberg metadata? One kind of metadata that Iceberg saves is a list of **manifest entries**. Each entry contains information about a data file (in our case, Parquet files) that is part of the table. Among other things, it contains the **path** of the file and the **size in bytes** of the file. Parquet is a very efficient format designed to allow engines to read the minimum amount of data required for their queries. At the end of the Parquet file, there's a **footer** containing information about the data in the file. This footer helps engines read only what they need or even skip the file completely. Data files are usually big, they contain the actual data! Therefore, query engines want to scan as few data files as possible, and as little data from each file as they can. The Parquet footer helps them accomplish this. To optimize performance, query engines like Spark don't immediately read the entire data file. First, they read only the footer to determine if the file is relevant to their query and which parts they should fetch. To locate this footer, the engine needs to know where in the file it begins. It finds this position using the file size information stored in the manifest entry of the Iceberg metadata. **The engine uses the size as specified in the manifest entry to locate the footer starting from the end of the file.** ### Back to the Issue In our case, since files were silently overwritten, **some metadata entries were wrong!** The old metadata entries contained incorrect information about file size. The actual file could be larger or smaller than what was recorded in the metadata. This discrepancy caused engines to look for the Parquet footer in the wrong place, explaining all the errors we encountered. - If the size in the metadata was **more** than the actual size - the engine requested an invalid range outside the bounds of the file - If the size in the metadata was **less** than the actual size - the engine tried to read the footer from the middle of the file instead of from its end, resulting in unparsable information Here is a simplified visualization of the information in the metadata, versus the actual data: ![](https://cdn.prod.website-files.com/68ef977c5ec5483ef208e8b3/68efac2d7195a84051d2f08a_68ca9fd5f92eb4e4dfbc7fb2_68779b3a20db6ad729efa786_screenshot3.avif) ## Wrapping Up The issue we encountered is not new. But despite a fix that was released 1.5 years ago, it is still widespread in production environments. - This is a severe issue. It caused data corruption, which rendered tables practically unreadable. - It is also a **data integrity issue**. When it manifested, **table data was permanently lost.** Without continuous monitoring of table metadata evolution, it is not even possible to know exactly which and how much data was impacted. - An Iceberg version containing this bug **is the default Iceberg version bundled with EMR 6.x**. Though EMR 7 has been available for some time, upgrading Spark versions is **hard**. As a result, EMR 6.x remains widely used. ### Are you Affected? - If you're using Spark Streaming with Iceberg ≤1.5.0, we recommend upgrading your Iceberg version. - If you're using EMR 6.x and cannot upgrade to EMR 7.x, you can still upgrade your Iceberg version. Rather than using the bundled Iceberg JAR, run your streaming applications with a newer version of Iceberg. You'll benefit from many other improvements too! ### Takeaways Iceberg gives you unprecedented ownership over your own data. With this freedom, however, comes responsibility that wasn’t there before. You are free to use whatever engine you want, but you will also have to deal with its own set of intricacies. You are free to save your data in whatever way that fits your need, but when it gets corrupted - it’s on you to debug and find a fix. ### Stay Tuned Next up in this series is a dive in into another serious Iceberg table corruption issue we encountered in the wild, stay tuned for [**Part 2**](https://www.ryft.io/blog/iceberg-table-corruption-and-data-loss-in-the-wild-part-2) **.** Table of Contents [Background](https://www.ryft.io/blog/iceberg-table-corruption-and-data-loss-in-the-wild-part-1#background) [The Problem](https://www.ryft.io/blog/iceberg-table-corruption-and-data-loss-in-the-wild-part-1#the-problem) [Data Loss and Table Corruption](https://www.ryft.io/blog/iceberg-table-corruption-and-data-loss-in-the-wild-part-1#data-loss-and-table-corruption) [Reading Iceberg Code](https://www.ryft.io/blog/iceberg-table-corruption-and-data-loss-in-the-wild-part-1#reading-iceberg-code) [Explaining the Fix](https://www.ryft.io/blog/iceberg-table-corruption-and-data-loss-in-the-wild-part-1#explaining-the-fix) [Bonus Section: Explaining Exceptions](https://www.ryft.io/blog/iceberg-table-corruption-and-data-loss-in-the-wild-part-1#bonus-section-explaining-exceptions) [Reading Parquets](https://www.ryft.io/blog/iceberg-table-corruption-and-data-loss-in-the-wild-part-1#reading-parquets) [Back to the Issue](https://www.ryft.io/blog/iceberg-table-corruption-and-data-loss-in-the-wild-part-1#back-to-the-issue) [Wrapping Up](https://www.ryft.io/blog/iceberg-table-corruption-and-data-loss-in-the-wild-part-1#wrapping-up) [Are you Affected?](https://www.ryft.io/blog/iceberg-table-corruption-and-data-loss-in-the-wild-part-1#are-you-affected) [Takeaways](https://www.ryft.io/blog/iceberg-table-corruption-and-data-loss-in-the-wild-part-1#takeaways) [Stay Tuned](https://www.ryft.io/blog/iceberg-table-corruption-and-data-loss-in-the-wild-part-1#stay-tuned) Get the latest posts straight to your inbox Thank you! Your submission has been received! ## Browse other blogs ![Apache Iceberg V3: Is It Ready?](https://cdn.prod.website-files.com/68ef977c5ec5483ef208e8b3/69415e4b45acad26189a17d2_Blog%20Coverv3.png) Engineering ## Apache Iceberg V3: Is It Ready? Apache Iceberg V3 is a huge step forward for the lakehouse ecosystem. The V3 specification was finalized and ratified earlier this year, bringing several long-awaited capabilities into the core of the format: efficient row-level deletes, built-in row lineage, better handling of semi-structured data, and the beginnings of native encryption. This post breaks down the major features, the current state of implementation, and what this means for real adoption. ![](https://cdn.prod.website-files.com/68ef977c5ec5483ef208e8b3/68f1061b22ca6c79303ddfab_Auteur_pic_01%20(3).avif) Guy Yasoor December 16, 2025 December 17, 2025 [blog](https://www.ryft.io/blog/apache-iceberg-v3-is-it-ready) ![Streaming with Apache Iceberg: The Operational Problems at Scale](https://cdn.prod.website-files.com/68ef977c5ec5483ef208e8b3/692895683e8be6007b3e1902_Blog%20Cover.png) Engineering ## Streaming with Apache Iceberg: The Operational Problems at Scale Streaming into Iceberg creates three operational problems most teams don't see coming: small files pile up faster than you can compact them, storage costs climb because you're paying for data you've already replaced, and merges take much longer than they should. After seeing these problems in real production data lakes, we decided to share more about the causes and possible solutions. ![](https://cdn.prod.website-files.com/68ef977c5ec5483ef208e8b3/68f105fe51b1bd3a206f8f26_Auteur_pic_01%20(2).avif) Yuval Yogev December 8, 2025 December 1, 2025 [blog](https://www.ryft.io/blog/streaming-with-apache-iceberg-the-operational-problems-at-scale) ![How to Choose an Apache Iceberg Catalog](https://cdn.prod.website-files.com/68ef977c5ec5483ef208e8b3/692068eb2d13507cc7245b1e_Blog%20Cover.png) Engineering ## How to Choose an Apache Iceberg Catalog Apache Iceberg has become the table format of choice for building open data lakehouses. It solves long-standing problems around ACID transactions and engine interoperability. This post covers how to choose an Iceberg Catalog for production. ![](https://cdn.prod.website-files.com/68ef977c5ec5483ef208e8b3/68f1061b22ca6c79303ddfab_Auteur_pic_01%20(3).avif) Guy Yasoor December 8, 2025 November 21, 2025 [blog](https://www.ryft.io/blog/how-to-choose-an-apache-iceberg-catalog) ## See Ryft in Action The only solution that automatically maintains and optimizes your Iceberg tables based on usage, no lock-in, no manual tuning [See Ryft in Action](https://www.ryft.io/see-ryft-in-action)