Latest AWS Data Structure Trends

AWS's announcement of S3 Tables and S3 Metadata isn't just another feature release - it's a strategic move in the data lakehouse space. Here's why this matters for CTOs & technology leaders: AWS's decision to build on Apache Iceberg is particularly interesting. Rather than pushing their own proprietary format or backing Apache Hudi (where they're the biggest contributor), they've aligned with what's becoming the de facto industry open table spec. ICEBERG ! This is a smart play that acknowledges a reality in modern data architecture: organizations want standards-based, interoperable solutions that don't lead to vendor lock-in and more of copy data syndrome. The performance claims are compelling - 3x faster queries and 10x more transactions per second. But what's more interesting is the automated maintenance features. The potential for 30-60% cost savings through automatic tuning isn't just about economics; it's about engineering efficiency. Your teams spend less time managing infrastructure and more time delivering value. Yes, you'll pay 15% more for storage and new fees for monitoring ($0.025/1000 objects) and compaction ($0.05/GiB). But let's be honest - what's more expensive: slightly higher storage costs or engineering time spent managing data optimization? For most organizations, the automated management will likely pay for itself in reduced engineering overhead. The hidden gem of S3 Metadata feature might seem less exciting, but it's solving a critical problem. Many organizations have built homegrown solutions to track what's in their data lakes. Having this capability built into S3 could eliminate a whole category of custom tooling that engineering teams have had to maintain. What Engineering Leaders Should Consider?? Adoption Strategy: If you're already using Iceberg, the transition path looks straightforward. If you're heavily invested in other table formats, you'll need to weigh the benefits against migration costs. Team Skills: While this simplifies many operations, it's still Apache Iceberg under the hood. Your team will need to understand Iceberg concepts to fully leverage these features. Cost Models: The pricing model rewards efficient data organization. It's worth reviewing your current data patterns to estimate the real cost impact. The integration with IAM for table-level RBAC, automated maintenance, and metadata management suggests a future where data lakes can be both powerful and manageable. For technical leaders, the key question isn't whether to use these features, but how to integrate them into your data strategy. The potential for reduced operational overhead and improved data governance makes this a compelling option, especially for organizations already heavily invested in AWS. One thing's certain: Iceberg won the open table format war, but the data lakehouse wars just got a lot more interesting. *Views are mine. #datastrategy #dataarchitecture #dataengineering #lakehouse #modernization #ai

Your data warehouse is a fancy restaurant—expensive, perfectly plated, but tiny portions. Your data lake, A farmers market—cheap, abundant, but chaotic and half the produce is rotten. Enter the Lakehouse: It's a food hall. Best of both worlds. For years, data teams were stuck choosing between warehouse reliability ($$$ per TB) or lake affordability (good luck finding clean data). The lakehouse revolution ended that tradeoff. 🏗️ What really Changed? Open table formats—Delta Lake, Apache Iceberg, Apache Hudi — all of these brought warehouse features to cheap cloud storage (S3, GCS, ADLS). Now you get: → ACID transactions on $20/TB storage (not $300/TB) → Time travel & rollbacks (undo bad writes instantly) → Schema evolution (add columns without breaking pipelines) → Unified batch + streaming reads Think: Database reliability. Cloud storage prices. Does this really make an Impact? Yes it does! → Netflix migrated petabytes from separate warehouse/lake systems to lakehouse—cut costs 40%, unified analytics. → Uber uses Delta Lake for 100+ petabytes—powers real-time pricing, fraud detection, all on one architecture. Curious to know When to Use What ❓ Lakehouse (Delta/Iceberg): → 90% of modern use cases → Large-scale analytics → Mixed batch + streaming workloads → Cost-conscious teams Pure Warehouse (Snowflake/BigQuery): → Small data volumes (<10TB) → Business analysts who live in SQL → Zero engineering tolerance Pure Lake (Raw Parquet): → Archival storage only → Need messy data Here are the Cloud Platforms solutions for Data Lakehouse: Amazon Web Services (AWS):  • S3 stores data; Glue, EMR process Delta Lake/Iceberg.  • Athena queries; Lake Formation governs access and auditing. Microsoft Azure:  • ADLS Gen2 stores data; Databricks runs Delta Lake.  • Synapse queries; Purview manages governance and compliance. Google Cloud:  • GCS stores data; Dataproc processes with Iceberg/Delta.  • BigQuery and BigLake query; Dataplex manages governance. Ready to level up? Which format are you exploring—Delta Lake or Iceberg? Drop your pick below! 👇

AWS just redefined how we think about object storage. Last week at re:Invent, Amazon announced the release of S3 Tables, starting with Iceberg. For those tracking the momentum behind open architectures and data catalogs, this is a significant move. Let’s unpack what it means. 1️⃣ AWS owning the storage layer: S3 is already the world’s most widely adopted object store. Moving up the stack into managing tables is a natural progression. AWS has unparalleled access to customer data needs, making this an intuitive evolution for their platform. 2️⃣ A unique implementation choice: Interestingly, AWS opted not to adopt the Iceberg REST API, which has become a standard for many data platforms. Instead, they added new APIs on table buckets that are what you’d expect for Iceberg table management. Unsurprisingly, they focused on tight integrations with their own analytics services like Athena, Redshift, and EMR rather than prioritizing integrations with external data platforms. This move reinforces their ecosystem play while enabling seamless cross-service functionality. 3️⃣ Governance and security made simple: With IAM policies already widely used across AWS deployments, extending them to S3 Tables streamlines governance and security. Access to tables is controlled through the same IAM policies we’ve all grown to know and love (or hate). 4️⃣ Operational simplicity: S3 Tables reduce the need for managing a 3rd party data catalog services. Features like built-in compaction and snapshot management further lighten the load for data engineers. This is all about making data infrastructure less operationally complex. 5️⃣ Iceberg first: The choice to focus on Iceberg over other open table formats speaks volumes about where the industry is headed. Iceberg’s traction is undeniable, and it wouldn’t be surprising to see AWS add support for other formats in the future. As the momentum behind open data architectures accelerates, major vendors are staking their claims, setting the stage for a race to the bottom on costs. Much like the commoditization of cloud storage over the past decade, this evolution is playing out one layer higher in the stack. Are S3 Tables a step forward for your use cases, or do you see gaps that still need to be filled? #AWS #reInvent #S3Tables #ApacheIceberg #DataEngineering #DataLakes #CloudComputing

AWS dropped a huge announcement yesterday that will have big ripple effects in the data industry. And in my opinion, it may have marked the death of Databrick's Delta Lake. So, what did they announce? A new service called Amazon S3 Tables. Under the hood, this is a brand new type of S3 bucket (called a "table bucket"), specifically optimized for storing data in Parquet and querying via Iceberg. You can think of the table bucket as your "database", and all the files stored in it will be "tables" -> hence "Amazon S3 Tables". The S3 Tables service will provide many services required to operationalize a data lake: table level permissions, metadata management, automatic file compaction/cleanup, and more. Why is this a big deal? Open data formats and data lakes have been all the rage over the past year. Many companies want to keep their data in their Cloud Storage provider and make it accessible to multiple services/query engines. AWS coming out and adding first class support for Parquet/Iceberg will lay down the foundations for this trend to accelerate. S3 Tables will become a new building block that many services (including Snowflake/Databricks) can and should build on top of. Now, back to Delta Lake... Delta Lake is the open source table format built & maintained by Databricks. It's an Iceberg alternative. Earlier this year, there were ongoing debates about what the best open source format is for your data lake. Iceberg and Delta Lake were the top two contenders. With AWS, the largest cloud provider, going out and building such a critical first class service centered entirely around Iceberg, they've gone out and stated very clearly: Iceberg is the winner. When a cloud giant this big throws all their weight behind Iceberg, people take notice. With this in mind, when given the choice between the two, who would bet on Delta Lake as their long term data lake file format that your whole company will build around? I certainly wouldn't. Exciting times.

Let’s do this! I speak to so many leaders and get so many insights into how the space is evolving! “Data 3.0 in the Lakehouse era,” using this map as a guide. Data 3.0 is composable. Open formats anchor the system, metadata is the control plane, orchestration glues it together, and AI use cases shape choices. Ingestion & Transformation - Pipelines are now products, not scripts. Fivetran, Airbyte, Census, dbt, Meltano and others standardize ingestion. Orchestration tools like Prefect, Flyte, Dagster and Airflow keep things moving, while Kafka, Redpanda and Flink show that streaming is no longer a sidecar but central to both analytics and AI. Storage & Formats - Object storage has become the system of record. Open file and table formats—Parquet, Iceberg, Delta, Hudi—are the backbone. Warehouses (Snowflake, Firebolt) and lakehouses (Databricks, Dremio) co-exist, while vector databases sit alongside because RAG and agents demand fast recall. Metadata as Control - This is where teams succeed or fail. Unity Catalog, Glue, Polaris and Gravtino act as metastores. Catalogs like Atlan, Collibra, Alation and DataHub organize context. Observability tools—Telmai, Anomalo, Monte Carlo, Acceldata—make trust scalable. Without this layer, you might have a modern-looking stack that still behaves like 2015. Compute & Query Engines - The right workload drives the choice: Spark and Trino for broad analytics, ClickHouse for throughput, DuckDB/MotherDuck for frictionless exploration, and Druid/Imply for real-time. ML workloads lean on Ray, Dask and Anyscale. Cost tools like Sundeck and Bluesky matter because economics matter more than logos. Producers vs Consumers - The left half builds, the right half uses. Treat datasets, features and vector indexes as products with owners and SLOs. That mindset shift matters more than picking any single vendor. Trends I see • Batch and streaming are converging around open table formats. • Catalogs are evolving into enforcement layers for privacy and quality. • Orchestration is getting simpler while CI/CD for data is getting more rigorous. • AI sits on the same foundation as BI and data science—not a separate stack. This is my opinion of how the space is shaping up. Use this to reflect on your own stack, simplify, standardize, and avoid accidental complexity!!!! ---- ✅ I post real stories and lessons from data and AI. Follow me and join the newsletter at www.theravitshow.com

If you’ve ever spent nights chasing delete files, compaction jobs, CDC glitches, or why is my upsert slow again? then you’ll understand why this AWS re:Invent update hit me personally. AWS just dropped some major support announcements for Apache Iceberg. Amazon Redshift now supports writing to Apache Iceberg tables. And the I one I had been waiting for- AWS support for Apache Iceberg V3 deletion vectors and row lineage. Apache Spark on Amazon EMR 7.12, AWS Glue, Amazon S3 Tables, Amazon SageMaker & the AWS Glue Data Catalog now support Iceberg V3 deletion vectors and row lineage. These solve the pain we don’t talk about enough: Deletion vectors = upserts & GDPR deletes without performance punishment Row lineage = honest, row-level audit history without duct-taped CDC logic According to me it's a big shift Lower storage costs, faster writes, comprehensive audit trails, and efficient incremental processing. And the best part is To take advantage of deletion vectors and row lineage in Iceberg V3, set the table property format-version = 3 during table creation. Refer to Apache Iceberg V3 on AWS for more details No rebuilds, no big effort to change. When you’ve operated lakehouses long enough, it’s a relief. You can learn more here:: https://lnkd.in/gbh-zt63

Last week, AWS made a big play with S3 Tables, bringing Apache Iceberg integration to their S3 Buckets. For those tracking the data landscape, this is pretty significant - it's AWS essentially picking their horse in the open table format race. The interesting part? This move basically transforms SageMaker from just an AI workspace into a full-blown platform where data and AI truly come together. Before, you had these point-to-point connections between different AWS services. Now? It's all unified under one hood. What's caught everyone's attention is how this positions Apache Iceberg against Delta Lake (Databricks' format). Despite Databricks' $1B acquisition of Tabular, they're playing nice - even working on ways to make the formats work together. One analyst put it perfectly: competing over table formats is like competing over TCP/IP - "it doesn't make a damn bit of difference." The real battle? It's shifting to the catalog level. That's where query engine providers can actually differentiate themselves. Nice to see the industry moving toward better interoperability, even if there are still some hurdles to clear. Follow Pratik Daga for more content on software engineering.

Feeling overwhelmed by AWS's latest data, analytics, and AI announcements from re:Invent? You’re not alone. The recent AWS re:Invent updates have been both exciting and staggering, showcasing a unified strategy across data, analytics, and AI. But with these innovations, there’s also been some confusion and angst here on LinkedIn about what it all means. https://lnkd.in/g57kP_Zz To help clarify, I’ve recorded a video breaking down AWS’s strategy, with a focus on: 💡 S3 Tables 💡 SageMaker Lakehouse / Unified Studio and Apache Iceberg compatibility 💡 Amazon Redshift I also dive into other key highlights, including: 🌟 Aurora DSQL 🌟 Neptune GraphRAG 🌟 New hardware and model announcements 🌟 Massive new Bedrock capabilities If you’re looking to make sense of these updates and understand their impact, this video is for you. I hope it helps demystify the rapidly evolving AWS landscape! #AWSreInvent #CloudComputing #data #datamanagement #dbms #cloud  #multicloud  #cloudnative #ai #futuretrends #aigovernance   #dataproducts #ML #LLM #agents #GenAI #GenerativeAI

🚀 𝗔𝗜 𝘁𝗼𝗼𝗸 𝗰𝗲𝗻𝘁𝗲𝗿 𝘀𝘁𝗮𝗴𝗲 𝗮𝘁 𝗔𝗪𝗦 𝗿𝗲:𝗜𝗻𝘃𝗲𝗻𝘁 𝘁𝗵𝗶𝘀 𝘆𝗲𝗮𝗿, but beyond the GenAI buzz, a 𝘨𝘢𝘮𝘦-𝘤𝘩𝘢𝘯𝘨𝘪𝘯𝘨 announcement for the data and AI industry may have flown under the radar: 𝗔𝗪𝗦 𝗦𝟯 𝗧𝗮𝗯𝗹𝗲𝘀 Why do I believe this is the most impactful announcement from re:Invent? 𝗪𝗵𝗮𝘁 𝗮𝗿𝗲 𝗦𝟯 𝗧𝗮𝗯𝗹𝗲𝘀? In simple terms, S3 Tables let users 𝗰𝗿𝗲𝗮𝘁𝗲, 𝘂𝗽𝗱𝗮𝘁𝗲, 𝗮𝗻𝗱 𝗾𝘂𝗲𝗿𝘆 𝘁𝗮𝗯𝗹𝗲𝘀 directly on S3, with relational-like functionality. Built on the Apache Iceberg open table format, it brings data correctness and consistency guarantees similar to traditional analytics database engines. 𝗜𝘀 𝘁𝗵𝗶𝘀 𝗻𝗲𝘄? Not entirely. Databricks pioneered similar capabilities with Delta Lake (a.k.a. the Lakehouse), and other platforms like Snowflake and BigQuery have recently followed suit. 𝗪𝗵𝘆 𝗶𝘀 𝗦𝟯 𝗧𝗮𝗯𝗹𝗲𝘀 𝘀𝗶𝗴𝗻𝗶𝗳𝗶𝗰𝗮𝗻𝘁 𝘁𝗵𝗲𝗻? S3 Tables democratize the open data lake architecture on one of the world’s most popular data storage systems: AWS S3. This means: 1. 𝗟𝗼𝘄𝗲𝗿-𝗰𝗼𝘀𝘁, 𝘂𝗻𝗶𝗳𝗶𝗲𝗱 𝘀𝗼𝘂𝗿𝗰𝗲𝘀 𝗼𝗳 𝘁𝗿𝘂𝘁𝗵 for analytics and AI data compared to other solutions. 2. 𝗦𝗲𝗮𝗺𝗹𝗲𝘀𝘀 𝗶𝗻𝘁𝗲𝗴𝗿𝗮𝘁𝗶𝗼𝗻 of AWS services and partner compute solutions, offering flexibility and value to customers. 𝗜𝗺𝗽𝗮𝗰𝘁 𝗼𝗻 𝗚𝗲𝗼𝘀𝗽𝗮𝘁𝗶𝗮𝗹 𝗔𝗻𝗮𝗹𝘆𝘁𝗶𝗰𝘀? Modern geospatial engines like Apache Sedona—designed for open data lake architectures—will thrive. Sedona can now process S3 tables data and read geospatial columns in WKB format. At Wherobots, we’ve developed an Iceberg extension for native geospatial support, which you can try Today on Wherobots Cloud (https://wherobots.com/). But, we are making geospatial data types natively accessible in the Iceberg standard and hence S3 Tables soon. Big things are coming—watch this space! 🌍📊 #AWS #reInvent #DataAnalytics #Geospatial #ApacheIceberg #ApacheSedona #AI 

Quick insight gleaned from Amazon Web Services (AWS) re:Invent just now. During a QA session with CEO Matt Garman, when he was asked to call out one of his favorite new announcements from the show, he zeroed in on S3 Vectors. Now generally available, S3 Vectors are basically a new, basic S3 object bucket type on AWS. Once you enable it, you get a set of APIs to store, read, and query vectors, all without provisioning any infrastructure. So no need to stand up a separate vector store like Pinecone or set up vector support within RDS via pgvector. This is a market trend we're following right now that's fueling a lot of innovation across the storage marketplace with vendors like Pure Storage, VAST, HPE, and many more are elevating object storage, pushing it up the app stack to encapsulate functionality that would normally sit inside a database. Already, AWS has done this for accessing Apache Iceberg tables natively from the object store via S3 Tables support, as another example. Why do this? Why turn the file system into a database? Cost efficiency for one. Object storage, which features capabilities like cold/warm tiers, can radically lower vector indexing and semantic search costs. Conversely, performance also comes into play here, as S3 can easily manage massive, TB-sized vector indexes at scale. This will make it easier for companies to vectorize much more information for use within LLMs and agentic processes. You can find more info on S3 Vectors here: https://lnkd.in/eWdHE9FM And here's a link to AWS' S3 Tables: https://lnkd.in/e3kSWvK5 The Futurum Group #AWSreInvent #AWS #Vectors #GenAI #AgenticAI