A lakehouse that is still healthy in year three has a small number of practices in place from the start: layered tables with derivation rules, partitioning chosen from real query patterns, maintenance jobs scheduled like production work, governance applied before workloads land, and an annual review that retires things.
None of this is exotic. It is the difference between a platform that improves and one that quietly degrades into a lake with better marketing.
What separates a lakehouse that lasts?
The failure is never sudden. Query times drift upward, one team writes directly to storage because the process was slow that week, a table nobody owns accumulates two years of data, and eventually users go back to extracts.
Every practice below exists to prevent one specific version of that drift. They divide into design decisions made once, and operational disciplines sustained continuously — and it is the second category that decides the outcome, because the first is usually done well by teams who are paying attention at the start.
Design practices
Layer the tables, and make the derivation rule explicit. Three layers — raw as ingested, cleaned and conformed, curated business models. Every table belongs to exactly one layer, and each layer is derived from the one above it by versioned code. The rule that carries the weight: no table is ever edited in place. If a correction is needed, it happens in the transformation and the downstream layer is rebuilt.
Keep raw immutable. The raw layer is what you have when a transformation turns out to be wrong. Teams that clean data on the way in save storage and lose the ability to reprocess.
Model for the questions, not for the sources. The curated layer should reflect business concepts, and its definitions should be the same ones approved in the business glossary — not a parallel set. Where those two drift, the organisation has two answers again.
Decide the direct-access question deliberately. Whether users can point their own tools at object storage, bypassing the engine and its access policies, is an architectural decision. Make it explicitly and document it, rather than discovering the answer during a security review.
Ingestion practices
Federate by default, ingest by exception. Every ingested copy is a copy to govern, secure and keep current. Ingest when there is a specific reason — query performance, history the source will not retain, or reducing load on a fragile system — and federate otherwise. The reasoning is developed in building a modern data platform.
Match the latency tier to the decision. Daily, minutes, seconds and sub-second have very different costs, and most requirements described as real-time are satisfied at minutes with change data capture. The tiers and their trade-offs are in real-time analytics: streaming, CDC and the lakehouse.
Make ingestion idempotent. Re-running a load must not duplicate rows. Keyed writes on a stable identifier, so a replay after an outage is safe rather than a data quality incident.
Contract the interface with source teams. The source system owner should know which fields are consumed downstream. This is the highest-value conversation available and it costs nothing.
Detect semantic drift, not just schema drift. Distribution checks on critical columns catch the status code with a new value and the field whose null rate jumped — changes that pass every schema validation and silently corrupt reports.
Table and storage practices
Partition for query predicates, not arrival order. The most consequential single decision. Use real query history to choose, which is an argument for giving analysts federated access before migrating storage.
Do not over-partition. A partition per customer per day produces millions of tiny partitions and a metadata problem on top of a small files problem. As a rule of thumb, aim for partitions large enough to hold hundreds of megabytes of data.
Target file sizes in the hundreds of megabytes. Ingestion naturally produces smaller files; compaction is what brings them back to a size queries can use efficiently.
Schedule compaction, snapshot expiry and orphan file cleanup as production jobs. With their own window and resource allocation, monitored like any other pipeline. This is the maintenance that nobody notices until it has been missing for six months.
Keep statistics current. The cost-based optimiser guesses without them, and a wrong join order on a large query costs orders of magnitude. Statistics collection belongs in the same maintenance schedule.
Sort or cluster large tables on a commonly-filtered column. It costs write throughput once and repays on every read through more selective file-level statistics.
Governance practices
Apply governance before workloads land. Classification, ownership and lineage on the lakehouse layer first. Retrofitting governance onto a populated platform is the most expensive avoidable mistake in this architecture.
Distinguish the technical catalog from the business catalog. The metastore tells engines where files are; the business catalog tells people what a table means and who owns it. Both are required. OvalEdge provides the second, and the distinction is set out in what a data catalog is.
Give every curated table a named owner. Not a team. The absence of a person is what allows a table to accumulate two years of unexamined data.
Enforce fine-grained access at the query engine. Object storage permissions are coarse; row-level and column-level policy belongs where it can be expressed once and applied across lakehouse tables and federated sources alike. This is Starburst's decisive capability for regulated institutions, as described in Starburst and Trino in Azerbaijan.
Make classification a machine-readable attribute. It has to drive masking, retention and — increasingly — what an AI system may retrieve. A classification that lives only in a policy document governs nothing.
Query and consumption practices
Separate interactive, scheduled and ad-hoc workloads. Resource groups or separate clusters, so one analyst's cartesian join cannot take down the morning dashboard load.
Cache the dashboard layer with a freshness window per table. Dashboards re-run identical queries constantly; that is the highest-return caching there is. Never cache what must be current.
Materialise the few aggregations that stay expensive, and review them annually so they do not accumulate into a permanent refresh tax.
Check pushdown before adding capacity. Most clusters that look under-provisioned are doing unnecessary work. The diagnostic order is in how Starburst accelerates analytics.
Define metrics once in a semantic layer. Otherwise the same metric is implemented three times in three BI tools and reconciling them becomes a quarterly ritual.
Operations practices
Name two platform engineers with the platform in their objectives. The single most reliable predictor of whether a lakehouse is healthy at eighteen months.
Write the runbook before go-live. Compaction, snapshot expiry, metastore maintenance, upgrades, capacity thresholds, and what to do when ingestion lag grows. A runbook written after the first incident is written under pressure.
Monitor three things separately: pipeline health, data quality and query behaviour. Teams conflate them and then cannot tell whether a bad number came from a failed job or a passing job with wrong logic.
Alert on lag, not just on failure. Streaming and CDC pipelines degrade rather than stopping. A pipeline that is four hours behind and still succeeding is producing confidently stale data.
Set a decommissioning date for whatever the lakehouse replaces, and treat that date as the deliverable. Dual-running indefinitely is the most common end state of these programmes, for reasons covered in lakehouse implementation challenges.
On-premise practices
Three additions for the deployments Yukon Labs runs in Azerbaijan.
Treat capacity as fixed and efficiency as the lever. A cloud team absorbs a bad partition strategy by scaling out; on-premise the same mistake is felt immediately and remedied through a procurement cycle. This raises the return on getting layout right early.
Own the object storage layer properly. MinIO or Ceph means erasure coding, rebalancing and disk failure handling are yours. Test a disk failure before production rather than during it.
Agree the update path for air-gapped environments at the start. Engine and connector updates arrive as artefacts through a controlled process. The first security patch is the wrong moment to design that process.
The reasons institutions here accept this operational burden rather than moving to a public cloud are set out in sovereign AI and, on the legal side, data residency and personal data law in Azerbaijan.
What should the annual review cover?
One practice that separates platforms that improve from platforms that accumulate. Once a year, with owners present:
Retire tables nobody queries. Query history tells you which. Every unused table is storage, maintenance and audit scope.
Retire materialised views whose queries are no longer run, and quality rules that have passed continuously for a year.
Revisit partitioning on the largest tables against the last twelve months of real query patterns, which have almost certainly changed.
Re-check ownership. People move. A table whose owner left is unowned regardless of what the catalog says.
Review retention. Data kept past its purpose expands the blast radius of every incident and the scope of every examination, a point developed in governance practices for regulated industries.
Key points
- Three layers with an explicit derivation rule, raw kept immutable, and no table ever edited in place.
- Federate by default and ingest by exception; make every load idempotent and contract the interface with source teams.
- Partition for real query predicates, avoid over-partitioning, target file sizes in the hundreds of megabytes.
- Schedule compaction, snapshot expiry and statistics collection as monitored production jobs.
- Apply governance before workloads land, give every curated table a named person, and enforce fine-grained access at the engine.
- Separate interactive, scheduled and ad-hoc workloads; cache dashboards with a per-table freshness window.
- Name two platform engineers, write the runbook before go-live, and alert on lag rather than only on failure.
- Review annually and retire things: unused tables, stale materialised views, quality rules that cannot fail, unowned assets.
Yukon Labs implements and operates lakehouse platforms on-premise in Azerbaijan with Starburst at the query layer and OvalEdge for governance. For the architecture, see data lakehouse architecture and building a modern data platform; to check where your estate stands, start with a readiness assessment.