Building a Modern Data Platform: A Reference Architecture

A modern data platform is eight layers: ingestion, storage with an open table format, transformation, query and serving, a semantic layer, catalog and governance, orchestration and observability, and an AI and retrieval layer. The architecture matters less than the order you build them in and which decisions you can reverse later.

This article is the reference architecture Yukon Labs works from in enterprise engagements, with the on-premise constraints that apply in this market built in rather than added afterwards.

What does a data platform actually have to do?

Before layers, four jobs. A platform that does these is adequate; one that does them elegantly but incompletely is not.

Make data findable. A user must be able to discover that a dataset exists, understand what it means, and learn who owns it, without asking a colleague.

Make it queryable without a project. New questions should not require a pipeline. When they do, the cost of curiosity rises over time and analysts stop asking.

Make it governed. Classification, access control and lineage applying uniformly, so that a sensitive column is sensitive in every layer it appears.

Make it reproducible. The same query should return the same answer tomorrow, and it should be possible to explain how any number was derived.

Every layer below exists to serve one or more of these. When a layer cannot be traced to one of them, it is usually there because a vendor sold it.

Layer 1 — Ingestion

Three ingestion modes, and most enterprises need all three. The mistake is standardising on one.

Batch extraction for systems that cannot support anything else — nightly extracts from legacy systems, file drops, third-party feeds. Unfashionable and still the majority of volume in most institutions.

Change data capture for operational databases where the analytical layer needs to stay close to current without polling the source. CDC reads the database transaction log and emits row-level changes, which is dramatically lighter on the source system than repeated queries. This is the mechanism that makes near-real-time analytics possible against a core banking system that cannot tolerate analytical load, and the detail is in real-time analytics: streaming, CDC and the lakehouse.

Streaming for genuinely event-driven sources — transaction events, telemetry, application logs.

And a fourth option that is often better than all three: do not ingest. If a source system can be queried in place through a federation engine, ingesting it creates a copy to govern, secure and keep current for no benefit. The default should be federation, with ingestion justified by a specific need: query performance, historical retention beyond what the source keeps, or reducing load on a fragile system.

That inversion — federate by default, ingest by exception — is the single largest difference between this reference architecture and the pipeline-centric designs most enterprises are running.

Layer 2 — Storage and table format

Object storage with an open table format on top. Two decisions, of different weight.

Object storage. S3-compatible, and on-premise for institutions here — MinIO and Ceph are the common choices. The properties that matter are horizontal scalability, cheap capacity, and an API that every engine already speaks.

Table format. Apache Iceberg, Delta Lake or Hudi. The format adds ACID transactions, schema evolution, time travel and partition management to what would otherwise be files in a bucket.

The table format decision deserves care because it is the most consequential reversible-in-theory decision in the platform: converting a large estate between formats is possible and unpleasant. Iceberg has become the default choice in most new enterprise builds on the strength of engine-neutral interoperability, with ongoing specification work extending the format. Delta is strong where the estate is Spark-centric. Hudi is strong on frequent upserts.

Two rules that hold regardless of choice:

Partition for how the data is queried, not for how it arrives. The most common performance problem in a new lakehouse is date-partitioned data queried by customer.

Plan compaction from the start. Streaming and CDC ingestion produce many small files, and small files degrade query performance steadily until someone notices. Compaction is a scheduled maintenance job, not an optional optimisation.

Layer 3 — Transformation and modelling

Raw data is not analysis-ready. Something must clean, conform and model it, and the architectural choice is where that runs.

Transform in place, in SQL, against the table format for most work. The engine that serves queries can also run transformations, which removes an entire tier of infrastructure.

Use a dedicated processing engine for heavy batch work. Spark remains the right tool for large-scale transformation, machine learning feature generation and anything requiring non-SQL logic. It coexists with the query engine rather than competing — they read the same tables.

Version transformation logic like application code. Transformations in a repository, code-reviewed, deployed through a pipeline. Transformation logic living in a scheduler's UI is the most common source of unreproducible numbers.

The layering convention worth adopting is three tiers: raw as ingested, conformed and cleaned, and curated business-facing models. The names matter less than the rule that each tier is derived from the one above it by versioned code, never edited in place.

Layer 4 — Query and serving

The engine that answers questions, and in this architecture it does two jobs at once: querying the lakehouse tables and federating to the systems that stayed where they are.

That second job is what makes the architecture achievable without a migration. The core banking system, the departmental databases, the systems whose owners will not agree to a move — the engine queries them in place and joins their results to lakehouse tables in one statement.

Starburst, the enterprise distribution of Trino, is the component Yukon Labs deploys for this, primarily because federated row- and column-level access control applies consistently across every connected source including those whose own security model cannot express it. The regional case is in Starburst and Trino in Azerbaijan, the engine comparison in Starburst vs Dremio vs Databricks SQL, and the performance mechanics — pushdown, caching, parallel extraction — in how Starburst accelerates analytics.

Serving splits into three patterns worth separating at design time: interactive BI queries, scheduled reporting, and application-facing APIs. The third is where teams over-engineer — a data platform is rarely the right backend for a customer-facing application with sub-second latency requirements.

Layer 5 — The semantic layer

The layer most enterprises skip and then rebuild after the third contradictory dashboard.

A semantic layer defines metrics once — what "active customer" counts, how revenue is recognised, which exclusions apply — and every consumer resolves the metric through it rather than reimplementing the logic in a BI tool.

Without it, the same metric is defined in the warehouse for one report, in the BI tool for another, and in a spreadsheet for the board pack. All three are defensible, they disagree, and reconciling them is a quarterly ritual.

The requirement it satisfies is reproducibility, and it is also where the platform meets governance directly: the metric definition in the semantic layer should be the same definition approved in the business glossary, not a parallel one. When those two drift, the organisation has two answers again, which is the problem the glossary was supposed to solve.

Layer 6 — Catalog, governance and access

Not a layer on top. A layer that intersects every other one.

The catalog holds what exists, what it means, who owns it, how it is classified and how it flows. It crawls the source systems, the lakehouse tables and the BI layer, and it is the only component with a complete view of the estate. OvalEdge is what Yukon Labs implements here; its architecture and connector model are described in how OvalEdge works.

Access control applies at two points that must not disagree: the engine enforces row- and column-level policy at query time, and the source systems enforce their own. The policy that matters is the one expressed once and applied consistently, which is why federated access control is a defining engine capability rather than a feature.

Classification is the attribute that drives everything else — masking, retention, residency, and what an AI system may retrieve. It has to be machine-readable, held on the asset, not written in a policy document.

Lineage is what makes change safe and audits tractable. Column-level, derived automatically from SQL and query logs rather than declared by hand.

The sequencing point that determines whether the platform is governable: this layer goes in early, before workloads land. Retrofitting governance onto a populated platform is the most expensive mistake available in this architecture, and it is the one most frequently made, for reasons set out in common data governance challenges.

Layer 7 — Orchestration and observability

Orchestration schedules and sequences the work: ingestion jobs, transformations, compaction, quality checks, catalog crawls. The requirement is dependency-aware execution with retry, alerting and a run history you can inspect — not a cron table.

Observability covers three distinct things that teams tend to conflate:

Pipeline health — did the job run, did it succeed, how long did it take, is it trending slower.

Data quality — do the outputs satisfy their rules. Completeness and validity on critical elements, routed by severity to someone who can decide what a failure means. The operating model behind this is in where data quality fits in a governance programme.

Query and cost behaviour — which queries dominate the cluster, which are scanning far more than they need, which users are affected when something degrades. On-premise this is capacity planning rather than a bill, but the analysis is the same.

Layer 8 — AI and retrieval

The newest layer, and the one that most changes the requirements on the layers beneath it.

An enterprise AI system needs four things from the platform:

Governed retrieval scope. Which documents and tables the system may reach, per user. This resolves to the classification attribute in layer 6, which is why AI deployments on ungoverned estates stall — the answer to "what can it see" does not exist yet.

The raw layer, not just the curated one. Retrieval works on documents, transcripts, logs and full-granularity history, which in a two-tier architecture live in the least governed place. Consolidating onto one governed copy is a large part of why the lakehouse and enterprise AI arrived on the same roadmaps, as covered in lakehouse architecture for enterprise AI and RAG.

Inference that respects residency. For supervised institutions here, inference runs inside the perimeter. The architecture, sizing and cost of that are in deploying LLMs on-premise, and the underlying constraint in data residency and personal data law.

An audit trail. What was retrieved, what was generated, who acted on it. This is a platform capability, not an application feature, and it is what an ISO/IEC 42001 audit asks to see.

What does this look like on-premise?

The reference architecture does not change for on-premise deployment; the component choices do.

Object storage becomes MinIO or Ceph rather than a hyperscaler service. The engine runs on Kubernetes in your data centre. The catalog and its repository run on your infrastructure. Inference runs on your GPUs. Nothing in the architecture requires internet connectivity at runtime, which is what makes air-gapped operation possible where it is required.

What genuinely changes is the operational burden. A managed cloud platform absorbs cluster operations, upgrades and capacity management; on-premise, someone owns them. This is a staffing question that should be answered before the architecture is approved, not after — and it is the most common reason an otherwise sound design underdelivers. The wider argument for why institutions here accept that burden is in sovereign AI.

In what order should it be built?

The sequence that minimises wasted work, from repeated engagements:

Phase 1 — Query and federation, over what already exists. Deploy the engine, connect the existing sources, give analysts federated access. This delivers value in weeks without moving any data, and it produces something more valuable than the capability itself: real query patterns showing what people actually ask.

Phase 2 — Catalog and governance. Classification, ownership and lineage across the sources connected in phase 1. Before any new storage layer exists, so governance is a precondition rather than a retrofit.

Phase 3 — Storage and table format. Stand up object storage and the table format. Migrate the workloads that measurably benefit — those with painful pipelines or history the source cannot retain.

Phase 4 — Transformation and semantics. Move transformation logic into version control, build the three-tier model, define metrics once in the semantic layer.

Phase 5 — Orchestration and observability. By now there is enough running to be worth orchestrating properly.

Phase 6 — AI and retrieval. On a governed, consolidated estate, with classification already driving retrieval scope.

The order is deliberate: each phase makes the next cheaper, and phases 1 and 2 are the ones that deliver value before any migration risk is taken. Programmes that start at phase 3 spend a year building storage nobody is querying yet.

Which decisions are hard to reverse?

Four. Everything else is a component swap.

Table format. Convertible, but at real cost across a large estate.

Partition strategy on your largest tables. Repartitioning a table with years of history is a project.

The classification model. Once assets are tagged and policies reference the tags, changing the scheme means re-tagging and rewriting policy.

Whether governance came first. Not a technical decision, and the least reversible of the four. An estate that grew ungoverned can be catalogued afterwards, but ownership, definitions and classification then have to be reconstructed from an estate nobody documented — which is what the data governance maturity model describes as being stuck at level 1 with a modern platform on top.

Engine choice, orchestrator, BI tool and inference stack are all replaceable. Design so that they are.

What team does this need?

Smaller than vendors imply, larger than a single platform engineer.

A working minimum for a mid-size institution: two platform engineers owning storage, engine and orchestration; two to three data engineers owning ingestion and transformation; one person owning the catalog and governance operations; and stewards embedded in the business domains at two to four hours a week each, as described in data stewardship roles and a working RACI.

The role most often missing is the sixth one — catalog and governance operations. Without it the platform runs and the estate quietly becomes undocumented again.

Key points

• Eight layers: ingestion, storage and table format, transformation, query and serving, semantic layer, catalog and governance, orchestration and observability, AI and retrieval.
• Federate by default and ingest by exception. Every ingested copy is a copy to govern, secure and keep current.
• Choose the table format deliberately; it is the most consequential decision that is only theoretically reversible.
• Partition for query patterns, and schedule compaction from day one.
• The semantic layer prevents the same metric being defined three times. Keep its definitions identical to the business glossary.
• Governance intersects every layer and must go in before workloads land.
• Build in order: federate, govern, then store, transform, orchestrate, and only then add AI.
• The role most commonly missing is catalog and governance operations, without which the estate becomes undocumented again.

Yukon Labs designs and implements this architecture on-premise in Azerbaijan, with Starburst at the query layer and OvalEdge at the governance layer. A readiness assessment establishes which phase your estate is actually at. For the architectural background, start with data lakehouse architecture, and for the practical obstacles, data lakehouse implementation challenges.