Data Engineering System Design: Scalability, Reliability & Trade-offs

Data Engineering system-design interviews test more than your knowledge of Kafka, Spark, Airflow, or cloud storage. They assess how you turn ambiguous business requirements into a scalable, reliable, and cost-aware architecture—and how clearly you explain the trade-offs.

A strong answer starts with requirements and workload assumptions, estimates scale, sketches the data flow, identifies failure modes, and explains how the system will be monitored and operated in production.

A reusable system-design framework

Clarify functional and non-functional requirements.
Estimate throughput, storage, retention, and peak load.
Define ingestion, processing, storage, serving, and orchestration layers.
Address correctness, retries, replay, idempotency, and recovery.
Explain scaling, observability, security, and cost controls.
State the trade-offs and what you would validate with a benchmark.

1. How do you approach an open-ended Data Engineering system-design question?

Short answer: Do not jump straight to a tool or architecture. Clarify what the system must do, quantify its workload, identify the most important constraints, then design the simplest architecture that meets them with a clear path to scale.

Step 1: Clarify requirements

Use cases: Is the system for dashboards, ML features, operational alerts, regulatory reporting, or all of these?
Sources: How many sources exist? Are they APIs, databases, application events, files, or CDC streams?
Scale: Events per second, average event size, daily volume, peak-to-average ratio, and growth rate.
Latency: Seconds, minutes, hourly, or daily? Is the target end-to-end or just processing time?
Correctness: Can records arrive late or out of order? Are duplicates possible? Must updates and deletes be reflected?
Retention and access: How long is raw data retained? Who queries it, at what concurrency, and at what latency?
Constraints: Budget, cloud environment, compliance, data residency, existing tools, and operational staffing.

Step 2: Define the high-level architecture

Sources
  ↓
Ingestion (batch, CDC, events, APIs)
  ↓
Durable landing / message log
  ↓
Processing (validation, enrichment, deduplication)
  ↓
Storage (raw → refined → curated)
  ↓
Serving (warehouse, lakehouse, OLAP, APIs)
  ↓
Dashboards, applications, ML features

Cross-cutting: orchestration, catalog, lineage,
security, data quality, monitoring, cost controls

This is a starting point, not a mandatory stack. A simple scheduled batch job may be better than a distributed streaming platform for a daily report. Conversely, a high-volume event stream with seconds-level freshness may need a durable broker and streaming processor.

Step 3: Explain the critical decisions

Why batch, streaming, or a hybrid model fits the latency requirement.
Where data becomes durable and how it can be replayed.
How duplicate events, late arrivals, schema changes, and failed jobs are handled.
How compute and storage scale independently.
How freshness, correctness, latency, and cost are measured.

Interview tip: State assumptions out loud. A defensible estimate based on explicit assumptions is more useful than an exact-looking number with no explanation.

Interview takeaway: Requirements first, capacity estimates second, architecture third, then failure handling and trade-offs. Keep the design tied to the business need.

2. How do you estimate the storage required for several billion events per day?

Estimate raw logical volume first, then adjust for encoding and compression, replication or redundancy, table-format overhead, indexes, intermediate datasets, and retention. Keep logical data volume separate from billed physical storage.

Worked example: 5 billion events per day

Assume 5 billion events/day and an average serialized event size of 1 KB. For an initial estimate, use decimal units: 1 KB = 1,000 bytes and 1 TB = 10¹² bytes.

Calculation	Result	Meaning
Daily raw volume	5,000,000,000 × 1 KB = 5 TB/day	Uncompressed logical payload estimate
30-day raw retention	5 TB × 30 = 150 TB	Before compression and overhead
90-day raw retention	5 TB × 90 = 450 TB	Before compression and overhead
Illustrative 3:1 compression	About 1.67 TB/day	Actual ratio depends on data and encoding

Add the details that matter in production

Event-size distribution: An average hides unusually large payloads. Sample actual records and account for schema and envelope overhead.
Compression and file format: JSON in a message broker may occupy more space than compressed Parquet or Avro on object storage.
Replication: A replicated message log can require multiple copies. Do not automatically apply the same replication multiplier to object storage, which may have its own durability model.
Derived datasets: Refined tables, aggregates, indexes, feature tables, and backfill copies can materially increase storage.
Retention tiers: Keep recent data in fast-access storage if needed and older raw data in a cheaper tier where access requirements allow.
Growth and peak load: Forecast volume growth and size ingestion and processing capacity for peaks, not only daily averages.
Temporary overhead: Compaction, rewrites, backfills, and concurrent snapshots may temporarily require additional space.

Useful formulas

Daily raw bytes = events per day × average bytes per event

Retained raw volume = daily raw bytes × retention days

Estimated physical storage = retained logical volume ÷ measured compression ratio + other retained datasets and overhead

Interview takeaway: Start with event count × average size × retention. Then estimate physical storage from real compression measurements, copies, derived data, and operational headroom.

3. How do you decide between batch and streaming processing?

The decision is driven primarily by how quickly consumers need correct data, how continuously the source changes, and whether the business value justifies the added operational complexity of streaming.

Dimension	Batch	Streaming
Latency	Usually minutes to hours or a scheduled interval	Can be seconds to minutes, depending on architecture
Processing model	Bounded input, often simpler to rerun	Continuous input, state and event-time concerns
Operations	Scheduling, dependencies, backfills, and batch retries	Lag, checkpoints, state, backpressure, and continuous availability
Typical use cases	Daily finance, warehouse transformations, historical recomputation	Fraud signals, live operational metrics, event-driven applications
Cost profile	May be efficient when work can run periodically	May need always-on compute and more operational investment

A practical decision process

Ask for the maximum acceptable end-to-end data delay—not just whether users say they want “real time.”
Determine whether the source provides reliable incremental changes, CDC, or event streams.
Assess whether the use case needs continuous state, event-time windows, or immediate actions.
Compare the cost and reliability of streaming with a frequent micro-batch or scheduled batch design.
Choose the simplest approach that meets the actual service-level objective.

Example: A daily executive finance report may be best served by a scheduled batch pipeline with reconciliation and rerun support. Fraud detection may require streaming because delaying the signal by an hour defeats its purpose.

A hybrid design is often practical: stream events into durable storage for immediate use, then run batch reconciliation to correct late arrivals, detect gaps, and rebuild historical aggregates.

Interview takeaway: Choose streaming for a genuine latency or continuous-processing requirement, not because it sounds more modern. Batch is often simpler for bounded, reproducible workloads.

4. How would you design a pipeline that can recover from failures without losing data?

Build the pipeline around durable checkpoints, replayable inputs, idempotent processing, transactional or atomic publication, and explicit recovery procedures. No single retry setting guarantees end-to-end correctness.

Core design principles

Make ingestion durable: Acknowledge events only after the required durability condition is met. Use a durable broker, landing zone, or source checkpoint as appropriate.
Track progress: Persist source offsets, batch watermarks, or processed time ranges so a restart knows where to resume.
Use idempotent writes: Assign stable event IDs or business keys and use deduplication or merge logic so replay does not create duplicates.
Separate processing from publication: Write to a staging location or new table version, validate it, and publish atomically where the storage system supports it.
Retry transient errors: Use bounded retries with exponential backoff and jitter for temporary network, throttling, or service failures.
Quarantine poison records: Send malformed or permanently invalid events to a dead-letter path with error details instead of repeatedly blocking the whole stream.
Preserve replay capability: Retain raw inputs and schema/version metadata long enough to reprocess a failed interval.
Reconcile: Compare source and target counts, keys, offsets, or business totals after recovery.

Example recovery flow

Read durable input from checkpoint N
        ↓
Transform and validate records
        ↓
Write idempotently to staging/target
        ↓
Commit or publish output successfully
        ↓
Persist checkpoint N+1
        ↓
Reconcile and emit observability signals

Checkpoint ordering matters. If the source checkpoint advances before the output is durable, a crash can cause data loss. If the output is committed but the checkpoint is not advanced, the same input may be replayed—so idempotent output is essential.

Exactly-once: be precise

Exactly-once processing is not a universal property of a pipeline just because a framework advertises it. End-to-end guarantees depend on source semantics, checkpointing, processing state, sink transactions, and how external side effects are handled. A common robust design is at-least-once delivery combined with idempotent writes and reconciliation.

Interview takeaway: Design for replay, make outputs idempotent, advance checkpoints safely, and test failures at every boundary—including after writing but before checkpointing.

5. How do you handle traffic spikes?

First distinguish a short burst from a sustained increase. Then determine where the bottleneck appears: source, broker, processing workers, network, storage, or serving layer. The architecture should absorb bursts without losing events and recover its backlog at a predictable rate.

Strategies for burst handling

Buffer with durable queues: A message broker or durable landing layer decouples producers from consumers and absorbs short-lived bursts.
Autoscale consumers: Scale based on backlog, lag, throughput, and processing latency—not CPU alone. Respect partition or parallelism limits.
Partition appropriately: Distribute load across partitions or shards while avoiding hot keys that overload one worker.
Use backpressure: Slow intake or upstream production when downstream systems cannot keep up, rather than overwhelming sinks.
Protect destinations: Use batching, connection pools, rate limits, and bounded concurrency to avoid overwhelming databases or APIs.
Separate workload classes: Prevent non-critical backfills from starving real-time pipelines through workload isolation or priority controls.
Plan backlog recovery: Estimate how quickly the system can process accumulated events after the peak ends.
Load test: Test expected peaks, sudden bursts, skewed keys, slow sinks, and partial outages.

Backlog recovery example

If incoming traffic is 100,000 events/sec and the system can process 150,000 events/sec after a spike, it clears backlog at a net 50,000 events/sec while normal traffic continues.

If the backlog is 9 million events, idealized recovery time is 9,000,000 ÷ 50,000 = 180 seconds (3 minutes), ignoring startup, variability, and downstream constraints.

Autoscaling cannot solve every bottleneck. If a single partition is hot, adding workers may not help until the key distribution or partitioning strategy changes. If the sink is the bottleneck, more consumers can make the situation worse.

Interview takeaway: Buffer, scale the actual bottleneck, apply backpressure, isolate critical workloads, and calculate backlog recovery—not just peak throughput.

6. What trade-offs would you make between latency, cost, and reliability?

These goals interact. Lower latency may require continuously running compute, more replicas, smaller batches, and more frequent writes. Higher reliability may require durable buffers, retries, redundancy, validation, and retained history. The right balance depends on the business impact of delayed or incorrect data.

Decision	Benefit	Cost or risk
Smaller, more frequent batches	Lower freshness lag	More scheduling, startup, and small-file overhead
More replicas or redundancy	Improved availability and fault tolerance	Higher compute, storage, and operational cost
Longer raw-data retention	Better replay, audit, and historical debugging	Higher storage cost and governance obligations
Stricter validation before publish	Reduces the risk of incorrect data reaching consumers	May increase latency or delay publication on failures
Shared compute pool	Higher average utilization and lower idle cost	Noisy-neighbor risk for critical workloads

Make trade-offs explicit with SLOs

Set a latency or freshness target for each data product based on business needs.
Define correctness and completeness expectations, not just availability.
Use error budgets or agreed tolerance for non-critical delays to avoid overspending on unnecessary speed.
Reserve capacity and stronger recovery guarantees for high-impact data products.
Measure total cost per workload or per unit of processed data, alongside latency and failure rates.
Revisit the design as volume, business usage, and incident patterns change.

Example: A fraud detection signal may justify low-latency streaming and redundant processing because delayed detection has high business cost. A monthly trend report may tolerate a daily batch and lower-cost storage because faster updates provide little extra value.

Interview takeaway: Do not optimize latency in isolation. Define service objectives by business criticality and spend where lower latency or higher reliability has measurable value.

7. How would you design a system that supports both real-time analytics and historical reporting?

Use a shared ingestion foundation and durable raw history, then serve different latency needs through appropriately designed processing and storage paths. Real-time consumers need fresh data quickly; historical reporting needs completeness, reproducibility, and consistent definitions across long time ranges.

Reference architecture

Applications / CDC / Event Sources
              ↓
       Durable Event Log
              ↓
      Raw Immutable Landing
          ↙          ↘
 Streaming Path     Batch / Historical Path
          ↓          ↓
 Real-time State   Refined Lakehouse / Warehouse
          ↓          ↓
 Live Dashboards   Curated Historical Models
          ↘          ↙
      Shared Metrics / Semantic Definitions

Cross-cutting: schema registry, catalog, lineage,
deduplication, quality checks, monitoring, access control

Key design decisions

One durable source of truth: Retain raw events or change records so data can be replayed and historical results rebuilt.
Separate serving paths: A streaming processor can maintain low-latency aggregates while batch jobs build complete historical tables.
Consistent business definitions: Share metric logic or semantic definitions to prevent real-time revenue and historical revenue from using different filters or grains.
Handle late and corrected data: Streaming results may be provisional. Use event-time windows, allowed lateness, and batch reconciliation or correction events.
Deduplicate across paths: If both paths process the same events, use stable event IDs and consistent rules to prevent double counting.
Control schema changes: Apply compatible contracts and coordinate changes across streaming and historical consumers.
Design for historical reproducibility: Preserve source offsets or snapshots, transformation versions, and enough raw history to rebuild affected periods.
Choose serving stores by access pattern: Use an OLAP store or low-latency aggregate for interactive real-time queries and a warehouse/lakehouse for broad historical analysis, where appropriate.

Example: e-commerce orders

A streaming path updates order count and gross sales every few seconds for an operations dashboard. A historical batch path joins orders, refunds, discounts, and late-arriving payment events to calculate finalized daily net revenue. The two views can differ temporarily, but the product should label provisional metrics and reconcile to a defined final result.

Interview takeaway: Share ingestion and data contracts, separate processing paths where latency demands it, define provisional versus finalized metrics, and reconcile streaming outputs with historical truth.

Quick Revision Cheat Sheet

System design: Requirements → scale estimates → architecture → failure handling → trade-offs.
Storage estimate: Events/day × average event size × retention; adjust for compression, copies, and derived data.
Batch vs streaming: Choose based on required freshness, source behavior, complexity, and cost.
Failure recovery: Durable input, checkpoints, idempotent writes, replay, bounded retries, and reconciliation.
Traffic spikes: Buffer, autoscale the bottleneck, use backpressure, and plan backlog recovery.
Latency vs cost vs reliability: Define SLOs by business criticality and measure the value of faster data.
Real-time plus history: Shared raw history, distinct processing paths, common metric definitions, and reconciliation.
Exactly-once: Verify the end-to-end source, processor, and sink guarantees; use idempotency where needed.

Scenario-Based Interview Practice

Try answering each scenario aloud in two to three minutes before expanding the suggested discussion points.

Scenario 1: Design ingestion for 5 billion events per day

Discuss: Estimate average and peak events/sec, event size, daily bytes, retention, and compression. Clarify freshness and consumer requirements. Design durable ingestion, partitioning, raw retention, processing, and monitoring. State assumptions and validate with load tests.

Scenario 2: The consumer falls behind during a major sale

Discuss: Check input rate, lag, per-partition skew, processing capacity, and sink throttling. Buffer events, scale within partition limits, isolate backfills, protect the sink, and calculate net backlog drain rate after the peak.

Scenario 3: A job crashes after writing output but before checkpointing

Discuss: Expect the input to be replayed. Use an idempotent sink, stable keys, transactional publication where supported, and checkpoint advancement only after durable output. Test this exact failure boundary and reconcile the result.

Scenario 4: Real-time and daily revenue do not match

Discuss: Compare metric definitions, time zones, event-time windows, late arrivals, refunds, deduplication, and source completeness. Decide whether the live metric is provisional, correct streaming logic where needed, and use historical reconciliation to produce a trusted final result.

Practice Challenge

Design a data platform for 2 billion events per day, with operational dashboards that need data within one minute and finance reports finalized by 8:00 AM daily. The platform must retain raw events for 90 days, recover from failures, and stay within a defined cloud budget.

What assumptions would you clarify before drawing the architecture?
How would you estimate daily storage and peak throughput?
Which parts would be streaming, batch, or shared?
How would you guarantee safe replay and avoid duplicate results?
How would you handle traffic spikes and slow destinations?
Which SLOs, dashboards, and cost metrics would you use to validate the design?

A strong answer makes assumptions explicit, uses estimates to guide architecture, and shows how correctness and recoverability are maintained as the system scales.

Search This Blog