Lakehouse implementations rarely fail on the architecture. They fail on eight operational realities: small files degrading queries, partitions chosen for how data arrives, confusion between two different things called a catalog, schema drift from sources, access control that stops at the storage bucket, concurrency surprises, dual-running that never ends, and no owner for the platform.

All eight are known, all eight are avoidable, and all eight appear in the second quarter rather than the first — which is why the pilot looks fine and the rollout does not.

Why does the pilot always look fine?

Because a pilot has one workload, a few tables, one team and a small volume of data. Every problem below is a function of scale, heterogeneity or time, and none of those exist in a pilot.

That is not an argument against pilots. It is an argument for treating the pilot as a test of the architecture and planning explicitly for the operational realities that only appear later. The build order that spreads this risk sensibly is in building a modern data platform.

Challenge 1 — the small files problem

The most common performance failure in a new lakehouse, and it is invisible until queries start crawling.

Streaming ingestion, frequent CDC micro-batches and per-partition writes each produce many small files. Query engines pay a fixed cost per file — opening it, reading its footer, planning around it — so a table split across 400,000 small files performs far worse than the same data in a few thousand right-sized ones, even though the bytes are identical.

The fix. Schedule compaction from day one, not when someone complains. Open table formats provide compaction and file-rewrite operations, and they need to be scheduled jobs in the orchestrator with the rest of the maintenance work. Target file sizes in the hundreds of megabytes rather than the kilobytes that ingestion naturally produces.

The trap within the trap. Compaction competes with ingestion for cluster capacity, so it needs a window and a resource allocation. Teams that schedule compaction without accounting for its cost discover their ingestion lag grows every night.

Challenge 2 — partitioning for how data arrives

The second most common, and the more expensive to correct.

Data arrives by date, so the table gets partitioned by date. Then the actual queries filter by customer, by branch, by product — and every query scans every partition. The engine's partition pruning, which is the largest single performance lever available, does nothing.

The fix. Partition for the predicates in real queries. This means knowing what those queries are, which is the practical argument for the build order that puts federated query access before storage migration: a quarter of real query history tells you how to partition, and guessing does not.

For tables with two competing access patterns, Iceberg's hidden partitioning and partition evolution reduce the cost of getting it wrong, since the partition scheme can change without rewriting every query. That flexibility is a reason to prefer the format, not a reason to skip the analysis.

What not to do: over-partition. A partition per customer per day produces millions of tiny partitions, which recreates the small files problem with a metadata explosion on top.

Challenge 3 — two different things are called a catalog

A genuine source of confusion in planning meetings, and worth separating explicitly because both are needed and they are not substitutes.

A technical catalog — the Iceberg or Hive metastore, or an equivalent — tells engines which tables exist, where their files are and what their current snapshot is. It is infrastructure. Without it the engine cannot read the table.

A business catalog — a data catalog in the governance sense — holds what a table means, who owns it, how it is classified and how it flows. It is an operating model tool. Without it the platform is undocumented.

Teams that hear "we already have a catalog" and mean the metastore end up with a technically functional lakehouse that nobody can navigate. The distinction and what the business layer actually delivers are set out in what a data catalog is; the platform component that provides it in these engagements is OvalEdge.

Challenge 4 — schema drift from the sources

The lakehouse table has a schema. The source system changes its own without telling anyone, because the source team has no reason to know the analytical layer exists.

Open table formats handle schema evolution gracefully — adding a column, widening a type — which is genuinely valuable. What they cannot handle is semantic drift: a status code whose meaning changed, a field that started being populated differently, a currency column that silently switched convention. The pipeline succeeds, the numbers are wrong, and nobody notices for a quarter.

The fix, in three parts.

Contract the interface. The source team should know which fields are consumed downstream. This is a conversation, not a technology, and it is the highest-value one available.

Detect drift automatically. Distribution checks on critical columns catch semantic change that schema checks miss — a status code with a new value, a field whose null rate jumped, a numeric column whose range shifted.

Route the change through impact analysis. Column-level lineage turns a source change into a visible list of affected reports before it ships rather than after, which is the mechanism described in governance practices for regulated industries.

Challenge 5 — access control that stops at the bucket

Object storage permissions are coarse: a principal can read a bucket or a prefix. Analytical access control needs to be fine: this user may see these rows, and this column masked.

Teams frequently discover this late, having assumed that storage-level permissions plus a query engine equals access control. What they get instead is all-or-nothing access to entire tables, and the workaround — creating filtered copies per user group — reintroduces the duplication the lakehouse was supposed to remove.

The fix. Enforce access control at the query engine, where row-level and column-level policy can be expressed and applied consistently across lakehouse tables and federated sources alike. This is Starburst's decisive addition over open-source Trino for regulated institutions, as covered in Starburst and Trino in Azerbaijan. Storage permissions then become coarse-grained protection for the engine's service identity rather than the user-facing control.

The related trap: direct storage access. If users can point Spark at the bucket, they bypass the engine and its policies. Whether that path exists is an architectural decision, and it should be made deliberately rather than discovered.

Challenge 6 — concurrency and the cluster you sized for the pilot

A pilot has five users. Production has two hundred, half of them running a BI tool that fires a dozen queries per dashboard refresh.

Query engines are memory-bound on joins and aggregations, and the failure under concurrency pressure is not graceful degradation — it is queries failing outright when memory is exhausted. Teams that sized the cluster on pilot workload discover this on the day the finance department connects.

The fix. Separate concerns rather than growing one cluster: resource groups or separate clusters for interactive queries versus scheduled reporting versus heavy ad-hoc analysis, so a single analyst's cartesian join cannot take down the morning dashboard load. Cache the repeated query patterns, which in BI workloads is most of them. And size for peak concurrency, which in practice means the fifteen minutes after the working day starts.

Challenge 7 — the migration that never finishes

The intended sequence is: build the new layer, migrate workloads, retire the old warehouse. The observed sequence is: build the new layer, migrate the easy workloads, run both indefinitely.

Dual-running has a real cost — two platforms, two sets of pipelines, two access models, and a permanent question about which system is authoritative. It happens because the last 20% of workloads are the hard ones, nobody owns their migration, and the old platform keeps working.

The fix, and it is organisational. Name the workloads to be migrated at the start, with an owner and a date each. Set a decommissioning date for the old platform and treat it as the deliverable rather than treating the new platform as the deliverable. And migrate the hardest workload third or fourth, not last — leaving it to the end guarantees it is attempted under deadline pressure with no remaining appetite.

Challenge 8 — nobody owns the platform

A lakehouse is more components than a managed warehouse: object storage, table format, metastore, query engine, orchestrator, catalog. Each is straightforward; the sum needs an owner.

The failure is a platform built by a project team that disperses at go-live. Six months later nobody has run compaction, the metastore has grown unhealthy, no one has upgraded anything, and query performance has degraded to the point where users have gone back to extracts.

The fix. Two named platform engineers with the platform in their objectives, a documented operational runbook covering compaction, snapshot expiry, metastore maintenance and upgrades, and monitoring that alerts before users notice. This is the sixth role that the reference architecture identifies as most commonly missing.

What is different on-premise?

Three additional realities in the deployments Yukon Labs runs here.

Capacity is finite and procurement is slow. Cloud teams absorb a bad partition strategy by scaling out. On-premise, poor query efficiency is felt immediately and adding hardware is a procurement cycle. This raises the return on getting partitioning and compaction right early.

Object storage is yours to operate. MinIO or Ceph rather than a managed service, which means erasure coding, rebalancing and disk failure handling are your operational responsibility.

Upgrades are a controlled process. In an air-gapped environment, engine and connector updates arrive as artefacts through whatever controlled path already exists for software. Agree that path at the start; the first time you need a security patch is the wrong moment to design it.

None of this argues against on-premise, which for most supervised institutions here is not a preference but a constraint — the reasoning is in sovereign AI. It argues for planning the operational side with the same seriousness as the architecture.

Key points

  • Every problem here is a function of scale, heterogeneity or time, so none of them appears in the pilot.
  • Schedule compaction from day one, with a window and a resource allocation of its own.
  • Partition for real query predicates. Federated query access before migration is what tells you what those are.
  • Distinguish the technical metastore from the business catalog. Both are required and neither substitutes for the other.
  • Detect semantic schema drift with distribution checks; schema checks miss it entirely.
  • Enforce fine-grained access control at the query engine, and decide deliberately whether direct storage access exists.
  • Separate interactive, scheduled and ad-hoc workloads before concurrency forces the issue.
  • Set a decommissioning date for the old platform and migrate the hardest workload early, not last.

Yukon Labs implements lakehouse platforms on-premise in Azerbaijan with Starburst at the query layer and OvalEdge for governance, including the operational runbook. For the architecture itself, see data lakehouse architecture and building a modern data platform; for the case that drives the move, why enterprises are moving to lakehouse architecture.