Data Lakehouse Solutions

Data pipelines are evolving from the "copy-paste" era to the "access-anywhere" era. Are you evolving with them? At its core, data pipelines answers one thing - How does data moves from "happened" to "decision made"? Here's how the evolution looks like: 𝗘𝗧𝗟 (The Old Way) Extract → Transform → Load  • Like preprocessing your groceries in the parking lot before bringing them inside  • Great for strict governance, legacy systems  • Slow. Rigid. Breaks on every schema change  • Tools: Glue, Talend, NiFi 𝗘𝗟𝗧 (The Cloud Era) Load raw → Transform in warehouse  • Load raw, transform in warehouse. Like, Dump groceries in fridge, prep later  • Scales with cloud compute  • Better, but still copying data everywhere  • Tools: dbt, Snowflake, BigQuery 𝗖𝗗𝗖 (Real-Time Tax) Change Data Capture  • Stream every database change via transaction logs  • Captures inserts/updates/deletes from source. Near real‑time, low overhead  • Perfect for fraud detection. Expensive for everything else  • Tools: Debezium, AWS DMS 𝗦𝘁𝗿𝗲𝗮𝗺𝗶𝗻𝗴 (Over-Engineering Champion)  • Real‑time processing, no waiting. Just like Food truck serving as you arrive  • Necessary for IoT, trading, Fraud and live dashboards  • Overkill for your Monday morning dashboard  • Tools: Kafka, Flink, Kinesis. Enter 𝗭𝗲𝗿𝗼 𝗘𝗧𝗟: The "Why Are We Doing This?" Moment The modern lakehouse game‑changer One city library for everyone – no need to photocopy books. Data lives in one place (lakehouse). Query it directly with any engine. No copying. 🤔 How Modern Lakehouses Make This Real? 𝗢𝗽𝗲𝗻 𝗧𝗮𝗯𝗹𝗲 𝗙𝗼𝗿𝗺𝗮𝘁𝘀 (Iceberg, Hudi, Delta) Think: Git for data. ACID transactions, time travel, schema evolution—on cheap object storage. 𝗦𝘁𝗼𝗿𝗮𝗴𝗲 ≠ 𝗖𝗼𝗺𝗽𝘂𝘁𝗲 Data sits in S3. Spark reads it. Snowflake queries it. Trino analyzes it. Same files. No copies. No vendor lock-in. 𝗡𝗮𝘁𝗶𝘃𝗲 𝗖𝗼𝗻𝗻𝗲𝗰𝘁𝗼𝗿𝘀 AWS Aurora → S3? Managed replication. No custom Glue jobs. No 3 AM debugging. What Changes for You? You stop: - Writing extraction code for the 47th time - Maintaining Airflow DAGs that "mysteriously fail on Wednesdays" - Explaining why reports are 6 hours delayed - Paying for storage in four different systems You start: - Designing smart partitioning strategies - Building dbt models where transformation actually matters - Focusing on data quality, not data plumbing - Solving business problems instead of pipeline problems The Real Talk? Zero ETL isn't "no data engineering."  It's engineering that matters. Traditional ETL won't vanish overnight—too much legacy infrastructure. But for new projects? Start lakehouse-first: - Store once in open formats - Let compute engines come to data - Create copies only when necessary Think of it this way:  You wouldn't photocopy a book every time someone wants to read a different chapter.  So why copy your data every time a new team needs access?

Forget about Schema Evolution; even schema enforcement is a major and problematic issue in a Data Lake. As a Data Engineer, we often discuss and use fancy methods to address Schema evolution, but I rarely see DEs focusing on Schema Enforcement in their pipelines. Most of the time, we simply dump or load data into a pure Data Lake without actually having a care for schema. There is a huge focus on keeping a schema while reading from the source, but while loading data to the target or sink, we hardly really care. If there is no schema enforcement, then multiple pipelines or changing versions & code of a single pipeline can actually keep adding multiple schemas to different Parquet or ORC files in the Partition or Non-Partition folder on Object storage. Without schema enforcement, there is nothing which can actually prevent this from happening. Readers (Spark, Presto, Trino, Hive) will infer a merged schema across files, which may lead to messy surprises. In a plain Data Lake setup — where you are just dumping data files (Parquet, ORC, CSV, JSON, etc.) into object storage (S3, GCS, ADLS), without a catalog, table format, or governance layer — there is no built-in schema enforcement. Lakehouse architecture and open table formats have attempted to resolve this issue and have contributed valuable knowledge to address this missing critical design aspect in simple Data Lake pipelines. Features such as Catalog, Governance layer, Data versioning, improved ACID support, better support for UPDATE/MERGE/DELETE, schema enforcement, and schema evolution are increasingly important due to serious Machine Learning pipelines and the necessity for reliable results in the Data Engineering landscape.

Some key insights from this week's chat with Justin Borgman, CEO of Starburst on The MAD Podcast / Data Driven NYC: 1) Database 101: * Transactional databases (Oracle, MongoDB) optimize for fast writes and consistency. * Analytical databases (Snowflake, Databricks, Starburst) focus on fast reads and complex queries to make sense of data 2) Data Lakes vs. Data Warehouses vs. Lakehouses: * Data lakes (originally Hadoop) allow massive storage but are inefficient for analytics. * Data warehouses (Snowflake, Teradata) optimize analytics but are proprietary. * Lakehouses combine both, leveraging open-source formats like Iceberg. 3) The Open Format War * Iceberg vs. Delta vs. Hudi: The industry debated which open table format would dominate. * 2024: The Year Iceberg Won: Snowflake embraced Iceberg, signaling broad adoption. Databricks acquired Tabular (the commercial Iceberg company) for $2B in a bidding war with Snowflake. Starburst had long supported Iceberg, making it a natural leader in the transition. 4) Starburst’s Strategy and Evolution * Justin co-founded the company with key contributors to Presto from Facebook * When Facebook claimed Presto, had to fork and rebuild a new open source project (Trino) * Building Starburst Galaxy (cloud product) twice: The first version didn’t deliver a seamless experience, leading to a costly but ultimately beneficial re-architecture. *Betting on the Lakehouse and Iceberg: Starburst has long championed the lakehouse model, believing open formats like Apache Iceberg would prevail. * Hybrid Solution: While Snowflake and Databricks are cloud-only, Starburst supports on-prem and hybrid deployments, making it attractive for enterprises with legacy infrastructure. 5) Starburst’s Offerings * Starburst Enterprise: On-prem self-managed solution * Starburst Galaxy: Fully managed SaaS version, optimized for cloud. * Dell Partnership: OEM deal embedding Starburst into Dell’s infrastructure, especially for AI workloads. 6) Core Capabilities: * Federated Query Engine: Runs analytics across multiple data sources. * Governance & Security: Fine-grained access controls, auditing, and data sovereignty compliance. * Streaming Data & Ingestion: Optimized for real-time analytics with Kafka. * Automatic Table Maintenance: Performance optimizations for Iceberg. 7) Starburst’s Role in the AI Stack Training AI Models: Access to high-quality, structured data is critical for AI. Retrieval-Augmented Generation (RAG): Providing AI with real-time structured data access for enterprise applications. 8) Go-to-Market Lessons * Starburst initially considered product-led growth (PLG) but realized its complexity required a direct enterprise sales motion. * Heavy reliance on solution architects to support large deployments. * Building a Services Ecosystem with boutique SI firms (e.g., Kubrick) to expand implementation support. * Strategic partnerships (Dell) took years to develop but became high-value revenue drivers.

I started working in data in the era of Hadoop. It was a great technical leap forward, but led to a damaging change in mindset that still persists today. Hadoop, and specifically, HDFS, freed us from the limitation of databases constrained by the size of the on-prem machine that hosted it, allowing us to store data at significantly reduced costs, and provided new tools to process this data that was now stored across many, low-cost machines. We called this a data lake. However, because storage became relatively cheap, we stopped applying discipline to the data we stored. We lowered the barriers of entry for data writers, harvesting as much data as we could. We stopped worrying about schemas when writing data, saying it's fine, we'll just apply the schema on read... → But by making writing data cheap, we made reading it 𝒆𝒙𝒑𝒆𝒏𝒔𝒊𝒗𝒆. For a start, it was almost impossible to know what data was in there and how it was structured. It lacked any documentation, had no set expectations on its reliability and quality, and no governance over how it was managed. Then, once you did find some data you wanted to use, you needed to write MapReduce jobs using Hadoop or, later, Apache Spark. But this was very difficult to do – particularly at any scale – and only achievable by a large team of specialist data engineers. Even then, those jobs tended to be unreliable and have unpredictable performance. 🙅♂️ Using this data became prohibitively expensive. So much so, it was hardly worth the effort. That's why people started calling them data swamps, rather than data lakes. Although some of us have moved away from data lakes, schema on read, etc, this mindset is still prevalent in our industry. We still feel we need writing data to be cheap. We still accept that a large portion of our time and money will be spent "cleaning" this data, before it can be put to work. But, if the data is as valuable as we say it is, why can't we argue for a bit more discipline to be applied when writing data? Why can't we apply schemas to our data on publication? We use strongly-typed schemas every other time we create an interface between teams/owners, including APIs, infrastructure as code, and code libraries. How much would costs be reduced for the company if that was the case? We can total up the time spent cleaning the data, the time spent responding to incidents, the opportunity costs of being unable to provide reliable data to the rest of the company. Is the ROI positive? I'd bet it is.

$150K and 3 months. That's what picking the wrong table format cost one team. Delta was perfect for their Spark workloads. But when they added Trino, Snowflake, and Athena to the mix, they needed Iceberg's multi-engine support. The lesson? Match your table format to your actual workflows. 𝐂𝐡𝐨𝐨𝐬𝐞 𝐀𝐩𝐚𝐜𝐡𝐞 𝐈𝐜𝐞𝐛𝐞𝐫𝐠 𝐰𝐡𝐞𝐧: → Your stack includes multiple engines → Partitioning strategy might change → You want vendor neutrality with REST standards → Schema evolution is frequent 𝐂𝐡𝐨𝐨𝐬𝐞 𝐃𝐞𝐥𝐭𝐚 𝐋𝐚𝐤𝐞 𝐰𝐡𝐞𝐧: → Spark/Databricks is your primary compute → You want Liquid Clustering (no more guessing partition columns) → You need fast UPDATEs/DELETEs via deletion vectors → Auto-compaction and less manual maintenance appeal to you 𝐂𝐡𝐨𝐨𝐬𝐞 𝐀𝐩𝐚𝐜𝐡𝐞 𝐇𝐮𝐝𝐢 𝐰𝐡𝐞𝐧: → Streaming ingestion with frequent upserts is your main workload → You need sub-minute data freshness → CDC pipelines are core to your architecture → You have massive tables with record-level indexing 𝐂𝐡𝐨𝐨𝐬𝐞 𝐋𝐚𝐧𝐜𝐞 𝐰𝐡𝐞𝐧: → Multimodal AI is your primary workload → You need vector search, full-text search, and random access → Feature engineering is core to your architecture → You want lakehouse benefits (ACID, time travel) with AI-native capabilities 𝐓𝐡𝐞𝐬𝐞 𝐟𝐨𝐫𝐦𝐚𝐭𝐬 𝐚𝐫𝐞 𝐜𝐨𝐧𝐯𝐞𝐫𝐠𝐢𝐧𝐠. Databricks acquired Tabular (Iceberg's creators). Delta UniForm generates Apache Iceberg metadata. Apache XTable translates between formats. And Lance brings AI-native capabilities while still integrating with open engines like Spark, Trino, and DuckDB. The format wars are cooling, but picking the wrong one today can still cost months of work. Match the format to your primary engine and workload pattern. Which camp are you in: multi-engine, Spark-native, streaming-heavy, or AI-first? #DataEngineering #ApacheIceberg #DataLakehouse Apache Iceberg Delta Lake DuckDB LanceDB

Open Source projects to watch for in 2026 (Data Infra space) Each and every year I hand-pick some of the Open Source projects that I personally think move the data infrastructure ecosystem forward. These are projects that shows real traction, solves genuine problems & influences how modern data platforms are being built. The important thing to note is that - Open source continues to be the backbone of today's data infrastructure! I consider myself very lucky to have had the opportunity to work with some of these projects in the last couple years. Here is my list (for 2026): ✅ Apache Iceberg: Iceberg continues to be THE open table format of choice for building lakehouse architectures. 2025 has been a huge year for Iceberg with uncountable organic adoptions and contributions from the community. Iceberg's simplicity & community continues to be its success factor. ✅ Apache Arrow: Arrow is everywhere. What started as an in-memory columnar format is now a multi-language foundation for high-performance data processing and data transport across systems. ✅ Apache DataFusion: DataFusion is an extensible query engine that uses Arrow as its in-memory format. 2025 has seen so many adoption of DataFusion - Arroyo, Comet, DBT-fusion, ParadeDB to name a few. ✅ Apache Hudi: Hudi isn't just a table format - it is an end-to-end open lakehouse platform with its robust storage engine. I cannot stress about the kind of innovation Hudi has brought into the lakehouse space - advanced indexes, non-blocking concurrency control methods, clustering/compaction as service. ✅ Apache Ozone: Ozone is a scalable, distributed object store designed for lakehouse workloads, AI/ML, and cloud-native applications. With S3-compatible APIs and a Hadoop-compatible filesystem, it’s a compelling open-source alternative for object storage. ✅ Lance by LanceDB: Lance is a lakehouse format for multimodal AI. It contains both a file format & a table format. I have seen some really good applications of Lance in the past year for vector search, full-text search & random access. ✅ Velox: Velox is a high-performance, open-source C++ execution engine built for reuse across batch, interactive, streaming & AI workloads. Velox has seen some really good adoptions like - Presto C++, Gluten, NVIDIA CuDF & I really like the community that is organically growing. ✅ Vortex: Vortex introduces a fresh take on columnar formats, focusing on fast random access to compressed data and zero-copy interoperability with Arrow. Its extensible design supports both general analytics and specialized embedded use cases. ✅ Apache Fluss (Incubating): Fluss is a streaming storage engine built for real-time analytics. It's built around the idea of "streams" as continuously updating tables and I think there’s a huge scope for low latency updates, real-time ingestion, etc. What other OSS projects should I keep an eye out for? #dataengineering #softwareengineering

“Data 3.0 in the Lakehouse era,” using this map as a guide. Data 3.0 is composable. Open formats anchor the system, metadata is the control plane, orchestration glues it together, and AI use cases shape choices. Ingestion & Transformation - Pipelines are now products, not scripts. Fivetran, Airbyte, Census, dbt, Meltano and others standardize ingestion. Orchestration tools like Prefect, Flyte, Dagster and Airflow keep things moving, while Kafka, Redpanda and Flink show that streaming is no longer a sidecar but central to both analytics and AI. Storage & Formats - Object storage has become the system of record. Open file and table formats—Parquet, Iceberg, Delta, Hudi—are the backbone. Warehouses (Snowflake, Firebolt) and lakehouses (Databricks, Dremio) co-exist, while vector databases sit alongside because RAG and agents demand fast recall. Metadata as Control - This is where teams succeed or fail. Unity Catalog, Glue, Polaris and Gravtino act as metastores. Catalogs like Atlan, Collibra, Alation and DataHub organize context. Observability tools—Telmai, Anomalo, Monte Carlo, Acceldata—make trust scalable. Without this layer, you might have a modern-looking stack that still behaves like 2015. Compute & Query Engines - The right workload drives the choice: Spark and Trino for broad analytics, ClickHouse for throughput, DuckDB/MotherDuck for frictionless exploration, and Druid/Imply for real-time. ML workloads lean on Ray, Dask and Anyscale. Cost tools like Sundeck and Bluesky matter because economics matter more than logos. Producers vs Consumers - The left half builds, the right half uses. Treat datasets, features and vector indexes as products with owners and SLOs. That mindset shift matters more than picking any single vendor. Trends I see • Batch and streaming are converging around open table formats. • Catalogs are evolving into enforcement layers for privacy and quality. • Orchestration is getting simpler while CI/CD for data is getting more rigorous. • AI sits on the same foundation as BI and data science—not a separate stack. This is my opinion of how the space is shaping up. Use this to reflect on your own stack, simplify, standardize, and avoid accidental complexity!!!! ---- ✅ I post real stories and lessons from data and AI. Follow me and join the newsletter at www.theravitshow.com

Last week, I came across two new table format projects — and it made me pause. So I went back to revisit the “open data stack” diagram I had once built live with another OSS founder, to see how things have evolved… and what’s still painful. Here’s what it looks like today 👇 🗂️ File formats: We often talk about Parquet and ORC as the backbone of the modern data lake. But reality is: many still ingest and store JSON, CSVs, and other semi-structured formats. And now, with AI workloads rising, we’re seeing new columnar and hybrid formats like Lance, Nimble, and Vortex — built for wide tables, random access, and unstructured data that Parquet can’t serve well. 📦 Table formats: Beyond the “big three” — Apache Hudi, Apache Iceberg, Delta Lake — we’re seeing experimentation again. Paimon brings an LSM-style storage layout, Lance blends blobs and vectors. New entrants like MSFT’s Amudai (already storing exabytes internally) and CapitalOne’s IndexTables, focusing on search and indexing, validate that there's still a lot of ground to cover here. 🔄 Interoperability: This is where the real work lies. Unification to “one format to rule them all” is a pipe dream — but interoperability is achievable. Projects like Apache XTable (Incubating) and DeltaUniform help bridge open formats and sync them into commercial engines (Onehouse, Databricks, EMR) and closed warehouses (Snowflake, Redshift, BigQuery). Let’s not forget: 70–80% of enterprise data still sits in closed formats today. 🧭 Open Metastores: We’re finally seeing viable Hive replacements — Polaris, UnityCatalog OSS, Gravitino, LakeKeeper. But there’s an important distinction: Open APIs (e.g. Iceberg REST) ≠ Open Metastore servers (operational catalogs). Most projects/vendors are building their own APIs to cover gaps, while exposing their tables as Iceberg using tools like XTable. We’re heading towards a messy N×M world of catalogs vs. formats. 🚧 Open problems: 1: Many Iceberg REST APIs are format-specific; we need multi-format APIs to future-proof interoperability. 2: Even beyond table formats, catalog data (policies, permissions, relationships) is still stored in proprietary databases — we need at least an open interchange format here. 3: The ecosystem still lacks actual, vendor-neutral, meritocratic bodies like the IETF or JCP community that can steward truly open data standards. So… what’s the takeaway? 👉 Unification is a myth. 👉 Interoperability is the path forward. 👉 OSS and users alike have their work cut out — to build a flexible, sustainable data ecosystem that allows innovation without fragmentation. #OpenData #DataLakehouse #ApacheHudi #XTable #Iceberg #DeltaLake #DataEngineering #OSS #Interoperability #TableFormats #Metastore #Data #BigData

𝗔𝗽𝗮𝗰𝗵𝗲 𝗜𝗰𝗲𝗯𝗲𝗿𝗴 𝗶𝘀𝗻'𝘁 𝗷𝘂𝘀𝘁 𝗮𝗻𝗼𝘁𝗵𝗲𝗿 𝘁𝗮𝗯𝗹𝗲 𝗳𝗼𝗿𝗺𝗮𝘁. 𝗜𝘁'𝘀 𝘁𝗵𝗲 𝗿𝗲𝗮𝘀𝗼𝗻 𝘁𝗵𝗲 𝗹𝗮𝗸𝗲𝗵𝗼𝘂𝘀𝗲 𝘀𝘁𝗼𝗽𝗽𝗲𝗱 𝗯𝗲𝗶𝗻𝗴 𝗮 𝘁𝗵𝗲𝗼𝗿𝘆 𝗮𝗻𝗱 𝗯𝗲𝗰𝗮𝗺𝗲 𝗽𝗿𝗼𝗱𝘂𝗰𝘁𝗶𝗼𝗻 𝗶𝗻𝗳𝗿𝗮𝘀𝘁𝗿𝘂𝗰𝘁𝘂𝗿𝗲. Before Iceberg, data lakes stored files. Just files. No transactions, no schema enforcement, no way to safely update a row without rewriting entire partitions. The source of truth was a folder listing slow, fragile, and expensive at scale. Iceberg changed the contract between storage and compute by making table metadata a first-class layer. 𝗪𝗵𝗮𝘁 𝗜𝗰𝗲𝗯𝗲𝗿𝗴 𝗮𝗰𝘁𝘂𝗮𝗹𝗹𝘆 𝗱𝗼𝗲𝘀: → 𝗔𝗖𝗜𝗗 𝘁𝗿𝗮𝗻𝘀𝗮𝗰𝘁𝗶𝗼𝗻𝘀 𝗼𝗻 𝗼𝗯𝗷𝗲𝗰𝘁 𝘀𝘁𝗼𝗿𝗮𝗴𝗲 Multiple writers can safely update the same table. No more corrupted reads during writes. → 𝗛𝗶𝗱𝗱𝗲𝗻 𝗽𝗮𝗿𝘁𝗶𝘁𝗶𝗼𝗻𝗶𝗻𝗴 The table format handles partition layout automatically. Consumers query columns they never need to know how data is physically organized. → 𝗧𝗶𝗺𝗲 𝘁𝗿𝗮𝘃𝗲𝗹 𝗮𝗻𝗱 𝘀𝗻𝗮𝗽𝘀𝗵𝗼𝘁 𝗶𝘀𝗼𝗹𝗮𝘁𝗶𝗼𝗻 Every commit creates an immutable snapshot. Query data as it existed at any point. Roll back bad writes without downtime. → 𝗦𝗰𝗵𝗲𝗺𝗮 𝗲𝘃𝗼𝗹𝘂𝘁𝗶𝗼𝗻 𝘄𝗶𝘁𝗵𝗼𝘂𝘁 𝗯𝗿𝗲𝗮𝗸𝗶𝗻𝗴 𝗰𝗼𝗻𝘀𝘂𝗺𝗲𝗿𝘀 Add, rename, or reorder columns. Existing queries keep working because the metadata layer handles the mapping. → 𝗘𝗻𝗴𝗶𝗻𝗲 𝗶𝗻𝗱𝗲𝗽𝗲𝗻𝗱𝗲𝗻𝗰𝗲 Spark, Flink, Trino, Dremio, DuckDB they all read the same Iceberg table. No vendor lock-in. 𝗪𝗵𝘆 𝗲𝘃𝗲𝗿𝘆 𝗽𝗹𝗮𝘁𝗳𝗼𝗿𝗺 𝗻𝗼𝘄 𝘀𝘂𝗽𝗽𝗼𝗿𝘁𝘀 𝗶𝘁: Snowflake, BigQuery, and Redshift all support Iceberg-managed tables natively. That's not a feature announcement. That's a shift in how data platforms are being designed. Iceberg doesn't replace Parquet or your warehouse. It gives object storage the table semantics warehouses assumed all along and keeps them open. What's your team evaluating Iceberg, Delta Lake, or Hudi? ♻️ Repost to help others ➕ Follow Arunkumar for data engineering and integration insights #DataEngineering #ApacheIceberg #DataLakehouse

I’ve debugged pipelines that wrote 3 files, crashed midway, then wrote them again. Now there are 6 files in storage. Two versions of the same data. And the query engine has no idea which ones are valid. This wasn’t bad engineering. This was just how data lakes worked. Data lakes were designed for one thing: scale. Store everything. Handle structure later. The underlying model was called BASE, basically available, soft-state, eventually consistent. That last part is the problem. “Eventually consistent” means your pipeline failure leaves behind partial files. Orphaned data. Wrong row counts. And the system just… moves on. No rollback. No alert. Just silent corruption sitting in your storage layer. Data warehouses had the answer to this. ACID transactions. Atomicity, consistency, isolation, durability. If something fails, it rolls back cleanly. Your data stays intact. But data warehouses couldn’t handle internet-scale volume. They weren’t built for unstructured data, streaming workloads, or ML pipelines. You couldn’t have both. So for years, data teams picked their poison: Scale with no guarantees. Or guarantees with no scale. That’s the gap the lakehouse was built to close. One unified platform with the flexibility of a data lake and the reliability of a data warehouse. Delta Lake is the most widely adopted open source lakehouse format solving exactly this. And if you’re a data engineer building anything AI-adjacent right now, understanding how it works under the hood is not optional. This is Day 1 of my Delta Lake breakdown series. I’m going deep on how it actually works, not just what it is. Tomorrow: how Delta Lake was born on a ski trip. Genuinely, that’s the story. Follow so you don’t miss it. I post daily. If you’ve ever spent a night debugging a data swamp, repost this. Someone on your feed needs to see it. PS: I write a weekly newsletter for data engineers breaking into AI engineering. 15,000+ readers. Free. ↳ https://lnkd.in/gRsiQNwf