Why Enterprises Are Moving to Lakehouse Architecture

Enterprises are moving to lakehouse architecture for five reasons that arrived together: paying twice to store the same data, open table formats becoming production-grade, AI workloads needing both raw and curated layers, exit costs from proprietary warehouses, and residency rules that rule out consolidating in a foreign cloud.

None of these on its own would justify re-architecting a working platform. Together they explain why the question in most enterprise architecture reviews has shifted from whether to when.

What problem was the two-tier architecture actually causing?

The architecture most enterprises still run has a data lake for raw and semi-structured data and a data warehouse for curated analytics, with pipelines moving data from the first to the second.

It worked, and it has three structural costs that compound as the estate grows.

The same data is stored twice, once in the lake and once in the warehouse, in different formats, with different access controls and different retention behaviour. Storage is cheap; two copies of storage plus the pipelines connecting them are not.

The pipeline layer becomes the bottleneck. Every new analytical question that needs data not yet in the warehouse becomes a request to the data engineering team, and that team has a queue measured in weeks. The cost of answering a new question rises over time, which is the wrong direction.

Governance splits. A column classified as sensitive in the warehouse is often unclassified in the lake, because the two are governed by different teams with different tools. The estate then has one well-governed copy and one that nobody is quite sure about, which is exactly the failure mode described in common data governance challenges.

The lakehouse proposition is to keep one copy in open formats on object storage, and run warehouse-grade SQL, BI and machine learning directly against it. The architectural detail behind that claim is set out in data lakehouse architecture: warehouse vs lake vs lakehouse.

Driver 1 — the economics of the second copy

The financial case is usually presented as storage savings, which understates it.

Storage is the smallest component. The real costs are the pipeline engineering to keep the second copy current, the compute to run those pipelines, the incident load when they break, and the opportunity cost of the questions never asked because the wait was too long.

An enterprise with several hundred pipelines is typically spending most of its data engineering capacity on maintenance rather than on new capability. Removing the second copy removes the pipelines that feed it, and that engineering capacity is the return — not the storage line item.

The honest counterweight: a lakehouse does not eliminate transformation. Curated, modelled tables still need building. What changes is that transformation happens in place, in open formats, without a separate proprietary storage tier underneath it.

Driver 2 — open table formats became production-grade

The lakehouse concept is older than its viability. What made it practical was the maturation of open table formats — Apache Iceberg, Delta Lake and Hudi — which add to object storage the properties that previously required a warehouse: ACID transactions, schema evolution, time travel and efficient partition handling.

Iceberg in particular has become the interoperability point that the architecture depends on, with the v3 specification work bringing further additions to the format and broad engine support across the ecosystem. The practical consequence is that the same table can be read by multiple engines — one for interactive SQL, another for batch transformation, a third for machine learning — without copying it into each engine's native format.

That property is what actually separates a lakehouse from a lake with a query engine bolted on. Format choice is consequential enough to deserve its own analysis, which is why table format selection is treated separately from engine selection.

Driver 3 — AI workloads need both layers

This driver did not exist when the two-tier architecture was designed, and it is now the one moving budgets.

Analytical workloads want curated, modelled, governed tables. AI workloads want something different: raw documents, semi-structured logs, historical records at full granularity, and the ability to reprocess everything when an embedding model changes.

In a two-tier architecture these live in different places under different governance, and an AI project ends up reaching into the lake — the layer with the weakest classification and access control. That is how a retrieval system becomes an uncontrolled access path to data nobody classified.

A lakehouse puts both workloads on one governed copy. The curated tables and the raw layer sit under the same access control and the same catalog, which means an AI system's retrieval scope can be reasoned about. Why that matters for output quality as well as compliance is developed in lakehouse architecture for enterprise AI and RAG.

Driver 4 — exit cost and vendor concentration

Proprietary warehouse formats create an exit problem that is invisible until it matters. Data stored in a vendor's internal format can only be read by that vendor's engine; leaving means exporting everything, which is expensive enough that most organisations do not.

Open formats change the negotiating position. Data in Iceberg on object storage can be read by Trino, Spark and a range of other engines, which means the engine becomes a replaceable component rather than a permanent commitment. The comparison of engines available at that layer is in Starburst vs Dremio vs Databricks SQL.

For supervised institutions this is not only commercial. Exit planning and concentration risk are standard supervisory concerns, and an architecture where the data is in an open format on infrastructure you control is materially easier to defend than one where extraction requires vendor cooperation — a point that sits directly alongside the outsourcing expectations in CBAR IT and data requirements.

Driver 5 — residency makes centralisation unavailable

For banks and state institutions in Azerbaijan, the standard modern architecture — consolidate everything into a cloud warehouse — is frequently not an option, because personal data cannot simply be relocated. The transfer rules and their engineering consequences are set out in data residency and personal data law in Azerbaijan.

That constraint shapes the answer here. A lakehouse on object storage you operate — MinIO or Ceph rather than a hyperscaler's service — plus an engine that can query it and federate to the systems that cannot move, is an architecture that satisfies residency without giving up modern analytics. The federation half of that is covered in Starburst and Trino in Azerbaijan.

The international case for the lakehouse is efficiency. Here it is more often feasibility.

What does a migration actually look like?

Not a migration in the traditional sense, and this is the most useful practical point. Successful moves are incremental and additive.

Stand up the lakehouse layer alongside what exists. Object storage, an open table format, an engine. Nothing is decommissioned yet.

Move one workload with a real owner. Choose a workload that currently requires a painful pipeline and has someone who will notice the improvement. Reproduce it on the new layer and run both in parallel until the numbers reconcile.

Federate rather than migrate the systems that will not move. The core banking system stays where it is. The engine queries it in place. This removes the biggest source of migration risk — you are not required to relocate the systems that are hardest to relocate.

Retire pipelines as they become redundant, one at a time, with the engineering time released as the measured outcome.

Move governance first, not last. Classification and ownership applied to the lakehouse layer before workloads land on it, not after. Retrofitting governance onto a populated lakehouse is the same expensive exercise as retrofitting it onto a lake, and it is the most common regret in these programmes.

Big-bang migrations of the whole estate are where lakehouse programmes fail. The failure modes are catalogued in data lakehouse implementation challenges.

When is a lakehouse the wrong answer?

Four cases, and they are common enough to check for explicitly.

A single-source, high-concurrency serving workload. A thousand concurrent users querying one database with sub-second requirements is a well-indexed database's job.

A small, stable estate. An organisation with one operational system and a warehouse that meets its needs does not have the problem a lakehouse solves. The architecture earns its complexity at scale and heterogeneity.

No platform engineering capability. A lakehouse is more components than a managed warehouse: storage, table format, catalog, engine, orchestration. Someone has to own them.

When the actual problem is trust in the data. A lakehouse makes ungoverned data reachable faster. If definitions are contested and lineage is unknown, the constraint is governance, and the architecture change will not relieve it. That sequencing argument runs through where data quality fits in a governance programme.

Key points

• Five drivers converged: duplicate storage and pipeline cost, production-grade open table formats, AI workloads needing raw and curated layers together, exit cost from proprietary formats, and residency constraints.
• The economic return is released data engineering capacity, not storage savings.
• Open table formats make the engine a replaceable component, which changes both the negotiating position and the supervisory exit story.
• AI is the driver moving budgets now, because retrieval workloads otherwise reach into the least-governed layer of a two-tier estate.
• Migrate incrementally: stand the layer up alongside, move one owned workload, federate what will not move, retire pipelines as they become redundant.
• Apply governance to the lakehouse layer before workloads land on it.
• Wrong answer for single-source high-concurrency serving, small stable estates, organisations without platform capability, or as a substitute for governance.

Yukon Labs implements Starburst on-premise as the query layer over lakehouse and legacy sources alike, and OvalEdge where the same estate needs a governed catalog. For the architectural background, start with data lakehouse architecture; to test where your estate stands, the readiness assessment is the usual entry point.