Real-Time Analytics: Streaming, CDC and the Lakehouse

Real-time analytics in a lakehouse means one of three things with very different costs: minutes-fresh data from change data capture, seconds-fresh from stream processing, or sub-second from a serving layer that is not the lakehouse at all. Choosing the wrong tier is the most expensive mistake in this area, and most requirements described as real-time are satisfied by the first.

The engineering question is therefore not how to go faster. It is what latency the decision actually requires, and what that latency costs to sustain.

What does "real-time" actually mean in practice?

Four tiers, and naming them explicitly ends most of the argument in requirements workshops.

Daily. The overnight batch. Correct for regulatory reporting, trend analysis and most management reporting. Unfashionable, cheap and adequate for the majority of analytical work.

Minutes. Change data capture streaming row-level changes into the lakehouse continuously. Suitable for operational dashboards, fraud review queues and anything where a person acts on the number within the hour.

Seconds. Stream processing with windowed aggregation. Required for alerting, real-time risk scoring and anomaly detection on transaction flows.

Sub-second. A serving database — key-value or an indexed operational store — that a customer-facing application queries. This is not analytics; it is application infrastructure that happens to serve derived data.

The useful discipline is to ask what decision the number supports and how quickly a human or a system acts on it. A dashboard refreshed every ten seconds and reviewed twice a day is paying continuously for latency nobody uses. Conversely a fraud rule evaluated nightly is not a fraud rule.

Why change data capture rather than repeated queries?

The naive approach to fresher data is to query the source more often. For an operational system this is the worst available option, and it is worth being precise about why.

Repeated extraction queries compete with the transactional workload for the same resources. On a core banking system with strict business-hours performance requirements, this is exactly the load the DBA has spent years keeping off the box. It also cannot detect deletes reliably, and it produces a consistent snapshot only by accident.

Change data capture reads the database transaction log instead. The log is already being written for durability and replication; CDC tails it and emits an ordered stream of row-level inserts, updates and deletes. Debezium is the common open-source implementation, with connectors for the major relational databases.

Three properties make this the right mechanism for enterprise sources:

Low source impact. Reading the log is far cheaper than repeated table scans, which is what makes near-current analytics possible against systems that cannot tolerate analytical queries at all.

Deletes and updates are captured correctly, including the before-image, which matters for auditability and for reconstructing state at a point in time.

Ordering is preserved, so downstream state can be rebuilt deterministically.

The practical cost is coordination. CDC requires log access, a supported database version, and usually a configuration change on the source. In a bank this is a conversation with the DBA team and a security review, and it is the phase where CDC projects actually spend their time — not in the pipeline code.

What breaks when streams land in a lakehouse?

Three things, reliably, and all three are manageable if planned.

Small files. A stream writing continuously produces many small files, and query engines pay a fixed cost per file. Without compaction, query performance degrades steadily until someone investigates. The mitigation is a compaction job with its own schedule and resource allocation, as covered in lakehouse implementation challenges.

The underlying trade-off is direct: write more frequently for lower latency and get more small files; batch writes into larger commits and get better query performance with higher latency. There is no configuration that avoids the trade — there is only choosing the point on it deliberately, per table, based on the latency tier that table actually needs.

Merge cost on updates. CDC produces updates and deletes, not just appends. Applying them to a table means merge operations, which are more expensive than appends. Open table formats handle this correctly, and formats differ in how they optimise it — this is the main technical reason to prefer one format over another for CDC-heavy workloads, since format specifications continue to evolve on exactly these operations.

Late and out-of-order data. Events arrive late; sources replay after an outage. A table that assumes arrival order will produce wrong aggregates. Event-time processing with a defined lateness window, and idempotent writes keyed on a stable identifier, are the standard answers — and both are much cheaper to design in than to retrofit.

Do you need a streaming platform, or is CDC enough?

A useful distinction, because teams frequently buy a full streaming stack for a requirement CDC alone satisfies.

CDC into the lakehouse is enough when the requirement is fresher tables. The pipeline is: source log, CDC connector, table writer, compaction. No stream processing, no windowed state, no separate serving layer. This covers the majority of "we need real-time reporting" requirements in enterprises.

Stream processing is required when the computation itself is continuous: windowed aggregations, joins across two event streams, stateful pattern detection. Fraud scoring on a transaction flow needs it; a dashboard of today's transaction volume does not.

A separate serving store is required when latency budgets are sub-second and concurrency is high — a customer-facing balance display, not an analyst's dashboard.

The cost difference between these is large, and it is mostly operational rather than licensing: a stream processing platform with stateful jobs is a permanent operational commitment, and on-premise that commitment is yours.

How do you serve queries against continuously-changing tables?

Two mechanisms, and they compose.

Query the lakehouse directly through the engine. Open table formats give readers a consistent snapshot while writers commit, so analysts see a coherent view rather than a partially-written one. For minutes-tier freshness this is usually sufficient on its own.

Federate to the source for the last mile. Where a query needs data fresher than the CDC pipeline delivers, the engine can query the operational system directly and union the result with lakehouse history. This is a genuinely useful pattern: history from the lakehouse where it is cheap, the current tail from the source where it is small. Starburst does this in one statement across both, which is the same federation capability described in Starburst and Trino in Azerbaijan applied to a latency problem rather than a residency one.

Caching then matters more than it does for batch workloads, because dashboards re-run the same aggregations constantly. The mechanics of pushdown, caching and parallel extraction are in how Starburst accelerates analytics.

When is real-time not worth it?

Four cases where the honest answer is to stay on batch.

Nobody acts on the number faster than daily. The most common case by a distance. If the report is reviewed each morning, overnight refresh is the correct design.

The source cannot support CDC and the workaround is aggressive polling. Frequent extraction queries against a fragile system trade a real operational risk for a marginal freshness gain.

Data quality controls run in batch. Streaming data into a lakehouse faster than it can be validated means publishing unvalidated numbers faster. Where quality rules require cross-record checks, the validation cadence sets a floor on useful freshness — the operating model behind that is in where data quality fits in a governance programme.

Nobody will own the pipeline. Streaming infrastructure fails differently from batch: it degrades rather than stopping, and lag grows quietly. Without monitoring and an owner, a streaming pipeline becomes a source of confidently stale data, which is worse than a batch job that visibly failed.

What does this look like in a bank here?

The pattern Yukon Labs implements most often in supervised institutions is deliberately conservative, and it is shaped by the source systems rather than by the target architecture.

Core banking stays on CDC at minutes-tier freshness, with the log-based approach specifically chosen so analytical demand never touches the transactional workload. Getting log access approved is a security review, and it is scheduled as one.

Departmental systems stay on batch unless there is a decision that needs them fresher. Most do not.

Federation covers the last mile where a genuinely current figure is needed, rather than pushing every source to a lower latency tier.

Streaming is reserved for cases with a continuous computation — transaction monitoring, alerting — rather than being adopted as a default architecture.

The reason for the conservatism is that on-premise capacity is finite and procurement is slow. Every latency tier you commit to is capacity you hold permanently, and in a data centre you own, over-provisioning for unused freshness is a durable cost rather than a line on a monthly bill. How this fits the platform as a whole is in building a modern data platform.

Key points

• Name the latency tier explicitly: daily, minutes, seconds, sub-second. Most requirements described as real-time are satisfied at minutes.
• Use log-based CDC rather than repeated extraction queries. It is lighter on the source, captures deletes correctly and preserves ordering.
• The real CDC project cost is log access approval and source configuration, not pipeline code.
• Streaming into a lakehouse produces small files. Compaction with its own schedule and resources is mandatory, not optional.
• Design for late and out-of-order events with event-time processing and idempotent writes; retrofitting this is expensive.
• CDC alone is enough for fresher tables. Stream processing is only required when the computation itself is continuous.
• Federate to the source for the last mile rather than pushing every pipeline to a lower latency tier.
• Skip real-time when nobody acts faster than daily, when the source cannot support CDC, when validation runs in batch, or when no one will own the pipeline.

Yukon Labs implements CDC and federated serving on-premise with Starburst at the query layer and OvalEdge tracking lineage across the streaming path. For the architectural context, see data lakehouse architecture and data lakehouse best practices.