The Breakdown of the Open Lakehouse in 2026: Iceberg, Arrow, Polaris, Parquet, and Ossie, and How to Actually Build One

By Alex Merced, Head of Developer Relations at Dremio

If you have followed the data world for the past few years, you have heard the phrase "open lakehouse" often enough that it may have started to sound like a slogan rather than a system. It is a system, and a remarkably well-designed one, and 2026 is a good year to take it apart on the workbench and see exactly how the pieces fit, because the pieces have all reached a level of maturity where the architecture is no longer aspirational. It is shipping, in production, at enormous scale, and it is assembled almost entirely from open standards that no single vendor controls.

The open lakehouse is best understood not as a product but as a stack of open specifications, each solving one layer of the problem of turning cheap cloud storage into a governed, high-performance, multi-engine data platform. Five Apache projects form the load-bearing structure of that stack in 2026: Apache Parquet at the file layer, Apache Iceberg at the table layer, Apache Polaris at the catalog layer, Apache Arrow at the in-memory and transport layer, and Apache Ossie, the newest arrival, at the semantic layer. Understand those five, how each works, what role it plays, and how they interlock, and you understand the open lakehouse, whether you intend to build one yourself from open components or buy a managed platform like Dremio or Snowflake that assembles them for you.

So this article is the breakdown: a component-by-component tour of the five projects, their current state and their role in the whole, followed by an honest look at the two paths to actually running an open lakehouse, self-managed from the open pieces, or managed through a commercial platform, with the trade-offs of each. I write about this stack for a living, I co-authored the O'Reilly books on two of these projects, and I now work at Dremio, which was acquired by SAP and is being integrated into SAP's data platform, a development I think is genuinely worth watching and will say more about at the end. My biases are on the table. The architecture, I will try to present as it actually is, because its elegance does not need my salesmanship.

First, Why "Open" Is the Whole Point

Before the components, the thesis that ties them together, because it explains why this particular architecture won and why its openness is not incidental.

For most of computing history, a data platform was a single vendor's integrated stack: their storage format, their engine, their catalog, their tools, all proprietary, all coupled, and switching any layer meant switching all of them. This produced lock-in of the most binding kind, because your data itself was trapped in a format only that vendor's software could read. The open lakehouse dismantles that coupling by making each layer an open standard that any tool can implement. Your data lives in open file formats on commodity object storage that you own. Your tables are defined by an open specification that many engines can read and write. Your catalog speaks an open protocol. The result is that no layer holds your data hostage, and you can mix engines, swap tools, and change vendors without migrating your data, because the data was never in anyone's proprietary format to begin with.

This is the same principle I have watched play out at every layer of computing, from open operating systems to open source software to open protocols: when the substrate becomes an open standard, competition moves up the stack to the engines and experiences built on top, and users win because they can no longer be captured. The open lakehouse applies that principle to analytical data, and the five projects below are the open standards that make it work. Each one, notably, lives at a foundation, the Apache Software Foundation, whose governance ensures no single company controls it, which is what makes "open" a structural guarantee rather than a marketing claim. Keep that in mind as we go: the reason you can trust this stack with a twenty-year data strategy is that the specifications are governed by an institution designed to prevent any vendor, including the ones who contribute most, from capturing them.

How the Stack Came Together

It helps to understand that these five projects did not arrive as a planned suite. They emerged over more than a decade, each solving a problem the previous layer exposed, and their convergence into a coherent stack is a story of the open ecosystem building itself layer by layer.

The story starts around 2013 with Parquet, born from a collaboration to bring efficient columnar storage to the Hadoop world, giving the ecosystem a good file format at a time when data increasingly lived as files on distributed storage. Arrow followed in 2016, created by many of the same people who had learned from Parquet, to solve the in-memory and movement problem that a good file format alone did not address, standardizing how columnar data lives in memory so systems could stop paying the serialization tax. Iceberg emerged from Netflix in the same era and was donated to Apache, born from the pain of managing enormous tables as collections of files on object storage, where the old Hadoop-era approaches broke down, and it solved the table-format problem that Parquet, a file format, was never meant to solve.

For years those three were the core, and the ecosystem ran on them while the catalog problem festered: Iceberg needed a catalog to coordinate its commits and govern access, and the available catalogs were either limited or proprietary, which threatened to reintroduce lock-in at the catalog layer just as the format layers had escaped it. The Iceberg REST catalog protocol standardized the interface, and then Polaris, contributed to open source and brought into Apache, provided a full open implementation with governance, closing the catalog gap around 2024 and 2025. And the semantic layer was the last gap to be recognized as a standards problem rather than a per-tool feature, addressed only recently as Ossie, because it took the pressure of AI agents needing shared definitions to make the industry treat semantics as something that must be open and standardized rather than locked in each vendor's tool.

Seen as a sequence, the pattern is clear: each layer became an open standard once the layer below it was solid enough to expose the next problem, and once the pain of leaving that layer proprietary became acute enough that the industry chose to standardize it. The open lakehouse is not a product someone designed. It is an emergent architecture that the open ecosystem assembled over a decade, each piece contributed by companies that concluded, at each layer, that an open standard served them better than a proprietary advantage, which is the deeper reason to trust it: it was built, layer by layer, by the accumulated judgment of an industry choosing openness each time the choice arose.

Layer One: Apache Parquet, the File Format

Start at the bottom, with the format that actually holds your data on disk, because everything above it depends on the properties Parquet provides.

Apache Parquet is a columnar file format, meaning it stores the values of each column together rather than storing each row's fields together. This single design choice is the foundation of analytical performance, because analytical queries typically touch a few columns across many rows, and columnar storage lets an engine read only the columns a query needs while skipping the rest entirely. Parquet packs data into row groups, and within them column chunks, compressed and encoded, with a footer at the end of each file that acts as a map: it records the schema, the location of every column's data, and statistics like the minimum and maximum values in each chunk. That footer is what makes Parquet fast on cloud storage. An engine reads the small footer first, uses the statistics to skip chunks that cannot match the query, and then fetches only the specific byte ranges holding the columns it needs, turning what would be a massive scan into a few targeted reads.

In 2026, Parquet is in the busiest period of its history, absorbing a wave of enhancements driven by the demands of AI and modern analytics. Recent work added a variant type for semi-structured data, letting JSON-like data be stored efficiently and queried without full parsing, along with native geospatial types, and the community is actively working on redesigning the file footer to reduce its cost on very wide tables, importing advanced float-compression techniques from the research world, and debating how the format should evolve without fragmenting its enormous installed base. The role of Parquet in the stack is foundational and stable: it is the universal container, readable by essentially every data tool on earth, that holds the actual bytes. Its moat is ubiquity, a decade of exabytes written and thousands of implementations, and its evolution is a careful renovation rather than a replacement, because the whole ecosystem sits on top of it. When you build an open lakehouse, Parquet is almost certainly the format your data files are written in, and the layers above it exist to turn collections of these files into something more than a pile of files.

Layer Two: Apache Iceberg, the Table Format

A pile of Parquet files is not a table. Making it behave like one, with transactions, schema changes, and the ability to be modified safely while being read, is the job of the table format, and Apache Iceberg is the open standard that won this layer decisively.

The problem Iceberg solves is that object storage gives you files, but analytics needs tables: things you can reliably insert into, update, delete from, and query, with all the guarantees databases provide, even as many engines read and write concurrently and the underlying storage refuses to let you modify files in place. Iceberg solves this with a metadata layer that sits above the data files. It tracks, in a tree of metadata files, exactly which data files constitute the table at each point in time, along with statistics about each file. A change to the table, however many files it adds or removes, becomes visible in a single atomic operation: the swap of a pointer to a new metadata file, coordinated through the catalog. Readers always see a consistent snapshot of the table, never a half-finished write, and because old snapshots persist until cleaned up, Iceberg gives you time travel, the ability to query the table as it existed at any past point, and safe rollback.

This metadata approach delivers the properties that make the lakehouse viable. Query planning never has to list storage, which is slow and expensive, because the metadata already knows every file and its statistics, so engines prune away most of the data before reading a byte. Schema and partitioning can evolve without rewriting history, because the format tracks them by stable identifiers rather than physical layout. And row-level updates and deletes, historically the hardest thing to do on immutable files, work through mechanisms the format has refined over successive versions, with the current generation using efficient deletion vectors that mark changed rows compactly.

Iceberg's state in 2026 is that of the established standard around which the entire industry has converged. Its v3 specification, in production this year, added the variant type, richer change tracking, and geospatial support, and the community is deep into designing v4 to make streaming-rate writes cheaper and metadata planning faster. The decisive fact about Iceberg is ecosystem gravity: every major cloud offers managed Iceberg, both major analytical platform vendors read and write it, and even competing table formats now publish Iceberg-compatible metadata so their tables can be read by Iceberg's ecosystem, which is what winning an interoperability standard looks like. In the stack, Iceberg is the layer that turns storage into tables, and it is the anchor the other layers organize around: the catalog governs Iceberg tables, the semantic layer defines meaning over them, and the engines read and write them. If you build an open lakehouse in 2026, Iceberg is very likely the table format at its heart.

Layer Three: Apache Polaris, the Catalog

Tables need a catalog, something that knows what tables exist, where their metadata lives, who is allowed to touch them, and that coordinates the atomic commits that make Iceberg's transactions work across many engines. Apache Polaris is the open catalog that fills this role, and it represents the layer where governance enters the open lakehouse.

To understand why the catalog matters so much, consider what it does. When an engine wants to read or write an Iceberg table, it asks the catalog where the table's current metadata is, and when it commits a change, the catalog performs the atomic swap that makes the change visible, ensuring that two engines writing at once do not corrupt the table. The catalog is therefore the coordination point of the whole system, and because every access flows through it, it is also the natural place to enforce governance: who can read which tables, who can write, what each principal is allowed to see. Polaris implements the open Iceberg REST catalog protocol, which means any engine that speaks that protocol can use it, and it adds the governance layer on top: role-based access control, so permissions are managed centrally, and credential vending, so that instead of handing engines long-lived storage credentials, the catalog issues short-lived, narrowly scoped credentials per operation, dramatically shrinking the security surface.

Polaris's state in 2026 is that of a young project maturing quickly. It graduated to a top-level Apache project earlier in the year after being contributed to open source by Snowflake and developed with contributions from Dremio and others, and its releases through the year have added federation, the ability to govern tables that live in other catalogs like AWS Glue or Hive under one Polaris umbrella, along with expanding format support and enterprise features. Its significance in the stack is that it makes the open lakehouse governable and multi-engine at once: without an open catalog, you either accept a proprietary catalog that couples you to one vendor, or you lose the central governance that enterprises require. Polaris gives you both open access and central control, which is the combination the lakehouse needs to be enterprise-ready. In the stack, Polaris is the governance and coordination layer, the control point through which engines reach tables and through which security is enforced, and it is increasingly where the semantic layer's definitions will be governed too, which connects it to the newest component.

Layer Four: Apache Arrow, the In-Memory and Transport Layer

The four layers so far describe data at rest, on disk and in the catalog. Apache Arrow governs data in motion and in memory, and though it is the least visible layer to most users, it is arguably the connective tissue that makes the whole multi-engine vision performant.

Arrow is a standard for how columnar data is laid out in memory, and a set of libraries and protocols built around that standard. The problem it solves is subtle but enormous: historically, every data system had its own in-memory format, so whenever data moved between systems, between an engine and a tool, between two engines, across a network, it had to be serialized out of one format and deserialized into another, a process that consumed staggering amounts of computation for no analytical benefit. This serialization tax was a hidden drag on the entire data ecosystem. Arrow eliminates it by defining a single, efficient, columnar in-memory format that every system can share, so data can move between Arrow-speaking systems with zero conversion, sometimes with zero copying at all, just a handoff of memory.

The consequences ripple through the stack. When an engine reads Parquet, it decodes into Arrow memory, and because Arrow's layout is columnar and designed for modern hardware, the engine's vectorized execution runs fast over it. When data moves between systems, Arrow's protocols, including Arrow Flight for high-speed network transport, move it without the serialization tax. When a Python data scientist pulls query results, Arrow lets them land in a dataframe without conversion overhead. Arrow is the reason the multi-engine lakehouse does not drown in translation costs: it is the common memory language that lets all the different engines and tools share data efficiently. In 2026, Arrow marked its tenth anniversary as a mature, ubiquitous standard embedded in countless data tools, with its ADBC database-connectivity standard and its broad language support making it more foundational than ever, even as most users never interact with it directly. Its role in the stack is to be the invisible high-performance substrate: the format data takes the moment it is read and the language it speaks whenever it moves, ensuring that an architecture assembled from many independent open components performs as though it were one integrated system. Notably, Arrow's origins trace to the same lineage as Dremio, whose founders helped create it, which is part of why the performance philosophy of Arrow runs through the whole open lakehouse approach.

Layer Five: Apache Ossie, the Semantic Layer

The four layers so far give you governed, performant, multi-engine tables. What they do not give you is shared meaning: agreement on what the data means, what a metric like revenue or churn is defined as, and how business concepts map to the physical tables. Apache Ossie, the newest of the five and the one still in its earliest days, is the open standard aiming to fill that gap, and its arrival completes the stack in a way that matters enormously for the AI era.

The problem Ossie addresses is semantic fragmentation. In most organizations, the definition of a key business metric lives in many places, in a BI tool's model, in a dbt project, in analysts' heads, in various dashboards' queries, each subtly different, so that the same question produces different numbers depending on which tool you ask. This has always been a costly problem, and the rise of AI agents querying data made it acute, because an agent with no shared definition of a metric will improvise one, producing confident answers computed with logic no one endorsed. Ossie is a vendor-neutral specification for defining semantic models, the metrics, dimensions, and relationships that give data meaning, in a format any tool can read and write, so that a definition authored once is understood everywhere.

Ossie's state in 2026 is that of a very young project with significant momentum. It began as the Open Semantic Interchange, an initiative launched by Snowflake with a coalition that grew to include Dremio, dbt Labs, Salesforce, and dozens of others, shipped an initial specification under an open license, and entered the Apache Incubator in June as Apache Ossie, with converters already built to translate between its format and existing semantic dialects. It is early, the specification is at an initial version and the hardest problems are still being designed in the open, so the right posture toward it is to align and experiment rather than to bet a migration on it this quarter. But its role in the stack is clear and important: Ossie is the layer where data becomes meaning, sitting above the tables and increasingly governed alongside them in the catalog, providing the shared definitions that both human analysts and AI agents need to produce consistent, trustworthy answers. It is the final piece that turns a governed data platform into a governed knowledge platform, and its progress through incubation is one of the developments I am watching most closely, because it completes the open stack at exactly the layer the AI era made critical.

How the Five Fit Together

Having toured the components, it is worth assembling them into the whole, because the elegance of the open lakehouse is in how cleanly the layers stack.

Picture a query traveling down through the layers. A question arrives, expressed in business terms, and the semantic layer, Ossie's domain, translates it from a business metric into a precise query against physical tables using the governed definition. The catalog, Polaris, authenticates the request, checks that the principal is allowed to access the tables involved, and vends short-lived credentials for the storage. The table format, Iceberg, tells the engine exactly which data files are relevant, using its metadata and statistics to prune away everything the query does not need, and coordinates a consistent snapshot. The file format, Parquet, lets the engine read only the specific columns and byte ranges required from each surviving file. And Arrow is the format the data takes the instant it is read, flowing through the engine's execution and out to the consumer without serialization overhead. Five open standards, five layers, one clean path from business question to governed answer, with no proprietary format anywhere in the chain and no vendor able to hold the data hostage at any layer.

The beauty of this design is that each layer is independently replaceable and independently governed, yet they interlock into a coherent whole. You can change engines without touching the data, because the engines all speak the same open formats. You can change tools without migrating, because the standards are shared. And you can trust the whole with a long-term strategy, because every layer is governed by an open foundation rather than a vendor. This is the architecture that broke the decades-old coupling of proprietary data platforms, and understanding it as five interlocking open standards, rather than as any vendor's product, is understanding what the open lakehouse actually is.

Path One: Building a Self-Managed Open Lakehouse

Now the practical question: how do you actually run one of these? There are two paths, and the first is to assemble it yourself from the open components, which is genuinely possible in 2026 in a way it was not a few years ago, and which suits organizations with the engineering capacity and the desire for maximum control.

The self-managed path means you take the open pieces and operate them. Your data lands as Parquet files in Iceberg tables on object storage you control, whether a cloud provider's storage or your own. You run an open catalog, Polaris, to govern and coordinate, deploying and operating it yourself. You bring open query engines, and here the ecosystem is rich: engines like Trino, Spark, Flink, and others, all open source, all able to read and write Iceberg through Polaris, chosen for the workloads you need, batch, streaming, interactive. Arrow is embedded in these tools already, doing its work invisibly. And as Ossie matures, you layer in open semantic definitions. The entire stack can be assembled from open-source components, run on infrastructure you own, with no license fees to a platform vendor and complete control over every layer.

The advantages of this path are real and follow directly from its openness: maximum control, no vendor dependency, no per-query or platform licensing costs beyond your infrastructure, the ability to customize any layer, and the assurance that comes from owning your entire stack. For organizations with strong data engineering teams, strict sovereignty requirements, or a philosophical commitment to open source, it is a legitimate and increasingly practical choice, and the fact that it is possible at all shows how complete the open ecosystem has become.

The costs of this path are equally real, and honesty requires stating them. You are now operating a distributed system with many components, each of which needs deployment, monitoring, upgrading, tuning, and troubleshooting, and the operational burden is substantial. The performance optimizations that managed platforms provide, sophisticated caching, automatic table maintenance like compaction and cleanup, query acceleration, you must implement, source from other open tools, or do without. The integration work of making the components cooperate smoothly, and keeping them working as each evolves, is ongoing engineering. And the expertise required is significant: someone on your team must deeply understand each layer. Self-managing an open lakehouse trades vendor dependency and license cost for operational complexity and the need for in-house expertise, which is a good trade for some organizations and a poor one for others, and the honest question is whether your team's time is better spent operating infrastructure or using it.

Path Two: Managed Platforms, Dremio and Snowflake

The second path is to use a commercial platform that assembles the open lakehouse for you, operating the components, adding performance and convenience on top, and giving you the open architecture without the operational burden. Two prominent examples, and I will be transparent that I work for one of them, are Dremio and Snowflake, and it is worth understanding what a managed platform adds and what the trade-off is.

A managed lakehouse platform takes the same open standards, your data stays in Iceberg tables in open Parquet files that you own, governed through open catalog protocols, and operates the hard parts for you while adding capabilities that would be significant engineering to build yourself. The key point, and the reason this is still an open lakehouse rather than a return to proprietary lock-in, is that your data remains in open formats that you own and that other tools can read. You are buying the operation and enhancement of the open stack, not a proprietary cage around your data, which means you retain the exit option that is the whole point of open: if you leave, your data is already in open formats, readable by anything.

What these platforms add is substantial. They operate the catalog and the table maintenance, compaction, cleanup, optimization, so you do not have to. They add performance layers, Dremio's approach, for example, centers on a query engine built on Arrow with intelligent caching and query acceleration through materializations it manages autonomously, delivering interactive performance over the open tables that raw open-source engines require significant tuning to match. They add federation, the ability to query across the lakehouse and other data sources in one place. They provide governance interfaces, semantic layers, and increasingly AI-facing capabilities like serving governed data to agents through standard protocols. And they handle the integration, so the components cooperate smoothly without your team doing the plumbing. Snowflake approaches from its own heritage as a cloud data platform that has embraced Iceberg and open catalogs, and Dremio approaches from its heritage in query federation and Arrow-based performance over the open lakehouse, and both let you use the open architecture without operating it yourself.

The trade-off is the classic one: you gain operational simplicity, performance, and support, and you take on a platform relationship and its costs, with the key mitigation that because your data stays in open formats, the relationship is far less binding than the proprietary platforms of the past. The right choice between self-managed and managed, and between managed platforms, depends on your team's capacity, your performance needs, your budget, and your tolerance for operational complexity, and the genuine advantage of the open lakehouse era is that whichever you choose, your data is in open formats, so the decision is reversible in a way it never was when data platforms meant proprietary data.

A Concrete Assembly: What a Real Stack Looks Like

To make the two paths tangible rather than abstract, let me sketch what an actual open lakehouse looks like when assembled, because seeing the pieces named in one configuration clarifies how they relate.

Imagine a mid-sized company standing one up. Their raw and refined data lands as Parquet files organized into Iceberg tables, physically sitting in object storage in their own cloud account, which means they own the bytes outright and pay only commodity storage prices for them. They run a Polaris catalog as the control point: every engine that touches the data authenticates to Polaris, which checks permissions through role-based access control and vends short-lived credentials scoped to the specific operation, so no engine ever holds broad standing access to the storage. On top, they run one or more query engines against those governed tables, perhaps an interactive engine for analysts and dashboards, a batch engine for heavy transformations, and a streaming engine for real-time ingestion, all reading and writing the same Iceberg tables through the same catalog, all speaking Arrow internally so data moves between them and out to consumers without serialization drag. As they mature, they define their key business metrics once in an emerging open semantic format so that every consumer, human and AI, computes the same numbers.

Notice what this configuration provides. There is exactly one copy of the data, in open formats, serving every workload, rather than a copy per tool. The governance is central, enforced at the catalog for every access regardless of which engine made it. The engines are chosen per workload and swappable, because they all speak the same open standards. And nothing is captured: if the company wants to change any engine, add a new tool, or move to a managed platform, the data is already in open formats readable by anything, so the change touches the compute, never the data. That last property is the entire payoff, and it is worth pausing on, because it inverts the old model where changing your platform meant migrating your data. In the open lakehouse, the data is the stable foundation and the compute is the interchangeable part, which is exactly backwards from how proprietary data platforms worked, and exactly right.

The self-managed path means the company operates all of that themselves, the catalog, the engines, the table maintenance, the tuning. The managed path means a platform operates it for them over the same open formats. But the architecture, the five layers and their interlock, is identical either way, which is precisely why understanding the components matters regardless of which path you take: the open lakehouse is the same system whether you run it or someone runs it for you, and the components are the same open standards in both cases.

The SAP Development Worth Watching

I will close the practical section with a development I am close to and genuinely think people should watch, and I will flag my involvement plainly so you can weigh it accordingly.

Dremio, where I work, was acquired by SAP and is being integrated into SAP's data platform, within SAP Business Data Cloud under the Data and Analytics organization. What makes this worth watching, beyond my obvious interest in it, is the combination it represents. SAP sits at the center of an enormous share of the world's enterprise operational data, the systems that run businesses, and Dremio brings a mature open-lakehouse platform built on exactly the open standards this article describes, Iceberg, Arrow, Polaris, the whole stack. The prospect of that open-lakehouse capability becoming part of the SAP platform means the open lakehouse architecture reaching into the core of enterprise data in a way it has not before, bringing open standards to a world that has historically been more proprietary, and connecting the operational data that SAP holds to the open analytical lakehouse. How exactly Dremio's functionality becomes part of the SAP platform is still unfolding, and I think it is one of the more promising developments in the enterprise data space precisely because it brings the open architecture to an enormous installed base.

I am not going to oversell it, because it is still taking shape and I am hardly a neutral party. But if you want to watch how the open lakehouse moves from the domain of data-native companies into the heart of traditional enterprise IT, this integration is one of the clearest windows into that shift, and I would genuinely encourage keeping an eye on it. The best ways to follow the developments are to follow the Dremio company page on LinkedIn, and to follow me directly, on LinkedIn at /in/alexmerced or on any other network, all of which are listed in one place at AlexMerced.com. I write and speak about exactly this, the open lakehouse and where it is going, continuously, and this integration is a story I will be tracking closely.

Questions I Hear Most Often

Do I have to use all five components? No, and most lakehouses in production today use four of them, Parquet, Iceberg, a catalog, and Arrow under the hood, with the semantic layer often provided by a platform's own capability while the open Ossie standard matures. The five are the complete open stack, and you adopt the layers you need, though the lower four are close to essential to calling something an open lakehouse.

Is the open lakehouse only for huge companies? No, and the managed-platform path in particular makes it accessible at modest scale, while the falling complexity of the open tools makes even self-management more approachable than it was. The architecture's benefits, no lock-in, multi-engine flexibility, one governed copy of data, apply at every size, and the entry cost has dropped considerably.

How does this relate to the data warehouse? The open lakehouse increasingly is the warehouse, or subsumes it: the capabilities that once required a proprietary warehouse, transactions, governance, performance, now exist over open tables, which is why the warehouse vendors themselves have embraced Iceberg and open catalogs. The distinction between lakehouse and warehouse is blurring toward the lakehouse's open-format foundation.

What makes it safe to build a long-term strategy on these? The Apache Software Foundation governance, which ensures no single vendor controls the specifications, combined with the ecosystem gravity that has made these the standards everyone implements. The combination means the specifications will outlive any individual vendor's involvement and will not be captured, which is exactly the property a long-term data strategy needs.

Where does AI fit into this stack? Throughout, and increasingly at the top: the semantic layer gives AI agents the governed definitions they need for accurate answers, the catalog governs their access as principals with the same controls as human users, and the open formats mean AI tools can read the data directly. The open lakehouse is becoming the governed foundation that enterprise AI stands on, which is why the semantic layer's maturation matters so much.

Self-managed or managed, honestly? Honestly, it depends on whether operating distributed data infrastructure is a good use of your team's time, which for most organizations it is not, making managed platforms the pragmatic default, while organizations with strong platform teams, strict sovereignty needs, or a commitment to running their own stack can and do self-manage successfully. The open formats mean the choice is not permanent, which is the luxury the open lakehouse provides.

Closing Thoughts

The open lakehouse in 2026 is a mature, elegant, and genuinely open architecture, assembled from five interlocking Apache standards that each solve one layer of turning cheap storage into a governed, performant, multi-engine data and knowledge platform. Parquet holds the bytes, Iceberg makes them tables, Polaris governs and coordinates, Arrow moves everything without friction, and Ossie, newest and still emerging, gives it shared meaning. Understanding the stack as these five open standards, rather than as any vendor's product, is understanding why it broke the decades-old lock-in of proprietary data platforms and why it has become the foundation the entire industry, including the former proprietary holdouts, now builds on.

Whether you assemble it yourself from the open pieces or use a managed platform to operate it for you, the defining property remains: your data stays in open formats that you own, governed by open standards that no vendor controls, which means you are never captured and the architecture is always yours. That is the promise the open lakehouse delivers, and in 2026 it delivers it in production, at scale, which is why I have spent so much of my career on it.

I co-authored Apache Iceberg: The Definitive Guide and Apache Polaris: The Definitive Guide for O'Reilly, along with a book on architecting an Iceberg lakehouse and further titles on data engineering and agentic analytics, all of which go far deeper than any article can into the components this piece introduced. And to keep up with where the open lakehouse is going, including the SAP and Dremio integration I mentioned, follow the Dremio LinkedIn page and follow me at /in/alexmerced or on any network listed at AlexMerced.com.

Browse the full collection of my books on data and AI at books.alexmerced.com.