Feature Engineering & Feature Stores

1. How would you design a feature pipeline for a fraud detection system?

Answer: Design around the model's latency and risk requirements. Fraud detection commonly combines historical behavior with near-real-time transaction signals.

1. Ingest events: capture transactions, account changes, device signals, and reference data. Validate schemas and retain event IDs and timestamps.
2. Stream reliably: use a durable event platform such as Kafka or Pub/Sub; handle duplicates, retries, late arrivals, and malformed records.
3. Compute features: calculate transaction counts and sums over 5-minute, 1-hour, and 24-hour windows, distinct devices, account age, and historical risk indicators. Use streaming for low-latency features and batch for large historical aggregates.
4. Preserve training/serving parity: use governed feature definitions and point-in-time-correct historical joins.
5. Store appropriately: maintain historical feature values in an offline store and low-latency latest values in an online store.
6. Serve and monitor: retrieve online features, apply explicit missing/stale-value policies, score the transaction, and monitor latency, freshness, quality, drift, and outcomes.

Example features

• Transactions by account in the last 5 minutes.
• Total transaction value in the last 24 hours.
• Distinct devices used in the last 7 days.
• Time since the account's previous transaction.

Trade-off: a strict inference latency budget requires fast serving, but stale or missing data must not silently be treated as a valid zero. Agree on fallback behavior with risk and product teams.

2. What is training-serving skew, and how would you prevent it?

Answer: Training-serving skew occurs when features used during model training differ from the features available or computed at inference. Offline metrics may look strong while production predictions degrade.

Common causes include duplicated SQL/Python logic, different null defaults or categorical mappings, time-zone differences, stale online values, and training data that contains information unavailable at prediction time.

Prevention: maintain canonical, versioned feature definitions; reuse tested transformations where practical; create training data with point-in-time joins; log production feature values; and compare online/offline values on sampled entities. Add contracts for types, ranges, nullability, and units.

Feature parity checks:
- Same feature name and version
- Same units, null rules, and mappings
- Same event-time semantics
- Freshness within the agreed SLA
- Offline/online values within tolerance

Tip: a feature store helps govern definitions and storage, but does not automatically eliminate skew.

3. What is point-in-time correctness?

Answer: Every feature value in a training example must reflect only information that was available when the prediction would have been made. This prevents a model from learning from the future.

Suppose a transaction occurred at 10:00 AM. A transactions_last_24h feature must include only qualifying information available by 10:00 AM. An event received at 10:05 AM must not be used in a simulation of what the system knew at 10:00 AM—even if its source event timestamp is earlier.

• Event time: when the event happened.
• Availability/ingestion time: when the platform could use it.
• Feature timestamp: the time the value represents or was computed.
• Prediction timestamp: when the model would have made its decision.

Important nuance

Joining only on entity ID is insufficient. Select the latest eligible feature value available no later than the prediction timestamp, applying the feature's validity window and late-arrival policy.

Retain timestamps and historical versions as needed. Use point-in-time joins or event-time range joins, and test late-arriving events and updates explicitly.

4. How can data leakage occur during feature engineering?

Answer: Leakage happens when training uses information unavailable at real prediction time, or when test-set information influences model preparation. It inflates offline performance and often fails in production.

• Future information: using a later chargeback or investigation outcome to predict whether the original transaction is fraudulent.
• Incorrect time joins: attaching the latest customer profile to an old training example.
• Preprocessing leakage: fitting imputers, scalers, or feature selection on the entire dataset before splitting.
• Target-derived features: calculating target encodings using validation or test labels.
• Split contamination: putting duplicates or strongly related records across train and test in a way that overstates generalization.

Prevention: define prediction timestamps and label windows; build point-in-time features; fit preprocessing on training data only; use chronological splits for time-dependent deployment; and investigate features with suspiciously high predictive power. If labels arrive 30 days after a transaction, a model scoring at authorization time cannot use the later chargeback label as an input.

5. When would you use an offline feature store versus an online feature store?

Answer: Offline stores support large-scale historical datasets, training, backfills, and analysis. Online stores provide low-latency access to current feature values during inference. Many systems use both while keeping definitions and versions aligned.

Dimension	Offline store	Online store
Primary use	Training, backfills, analytics	Real-time inference
Data	Historical values	Latest or window state
Optimizes	Throughput, scans, cost	Low latency, predictable reads
Typical storage	Warehouse or lakehouse	Low-latency key-value database
Key concerns	Lineage and historical correctness	Freshness, availability, p95/p99 latency

For overnight batch scoring, an offline store may be enough. For an API that must score a payment in milliseconds, an online store is usually needed. Consider consistency, retention, cost, access control, volume, and operational burden; not every feature needs online materialization.

6. How would you compute rolling aggregates for real-time inference?

Answer: Use a stream processor with event-time windows or keyed state. Key events by entity (for example, account ID), maintain counts/sums for required windows, and publish updated values to the online serving layer.

For each transaction:
  key = account_id
  validate and deduplicate event
  update state for relevant time windows
  handle late events by defined policy
  publish feature values
  make values available for inference

• Window semantics: tumbling windows are fixed and non-overlapping; sliding windows represent a moving interval; session windows group activity separated by inactivity.
• Time semantics: event time often best represents behavior; processing time may suit operational metrics.
• State and scale: manage state size, TTL, hot keys, recovery, and rebalancing.
• Duplicates and late events: use event IDs, watermarks or equivalent progress tracking, and a correction policy.
• Serving: define behavior when updates are delayed or the online store is unavailable.

Exact distinct counts over long windows can be expensive. If acceptable to the business, sketch-based approximate algorithms can reduce state and compute costs; validate their error bounds for the use case.

7. How would you backfill historical features after a business logic change?

Answer: Treat the change as a versioned data migration, not a blind rerun over production data. Clarify whether the new logic applies only going forward or whether historical training examples must be recomputed.

1. Version the definition: document changed logic, dependencies, effective date, and expected impact.
2. Make backfill reproducible: use recoverable source data, correct time semantics, idempotent writes, and date/entity partitions.
3. Isolate output: write to a separate feature version or location instead of overwriting the production set.
4. Validate: compare counts, null rates, distributions, sample entities, boundary timestamps, and expected old/new differences.
5. Scale gradually: run a sample, then a bounded period, then the full history while controlling shared compute load.
6. Reconcile live updates: prevent older backfill output from overwriting newer streaming values.
7. Publish safely: switch consumers after validation and retain the previous version for rollback.

Production-safe pattern

Create a shadow feature version, validate it, then switch readers through a controlled configuration or deployment. Keep a rollback path and record which feature version each model uses.

A business rule may have been valid only from a specific date. Applying today's rule to all history can misrepresent the past. Clarify whether the goal is historical truth, consistent training, or counterfactual reprocessing under the new rule.

8. How would you monitor feature freshness and distribution drift?

Answer: Monitor both whether features arrive on time and whether their values change unexpectedly. Freshness is primarily an operational signal; distribution drift is a data/model-risk signal. They are related but not interchangeable.

Monitor	Example metric	Alert example
Freshness	Current time minus latest feature timestamp	Feature age exceeds SLA
Pipeline health	Lag, failures, processing delay	Consumer lag spikes
Completeness	Null rate and entity coverage	Missingness exceeds baseline
Distribution	Quantiles, PSI, KS statistic, category frequencies	Sustained shift beyond validated threshold
Serving	Lookup errors, p95/p99 latency, fallback rate	Serving SLO breached

Set per-feature freshness SLAs and expected ranges; emit metrics by feature version and relevant segment; compare production distributions to an appropriate reference window; and link alerts to runbooks. For supervised models, also monitor delayed outcomes and model quality—feature drift alone does not prove that performance has degraded.

When freshness breaches its SLA, inspect upstream source health, consumer lag, processing failures, and online-store writes. Quantify affected decisions and invoke the agreed fallback. For drift, investigate source changes, seasonality, business shifts, and deployments before deciding whether retraining is appropriate.

Quick Revision Table

Concept	Key takeaway
Fraud pipeline	Reliable ingestion, streaming/batch features, offline history, online serving.
Training-serving skew	Align definitions, transformations, timing, and defaults.
Point-in-time correctness	Use only information available at prediction time.
Leakage	Exclude future outcomes and test-set information.
Offline vs online	Historical throughput vs low-latency serving.
Rolling aggregates	Define windows, event time, state, deduplication, and late-event policy.
Backfills	Version, isolate, validate, reconcile live writes, and retain rollback.
Monitoring	Track freshness, lag, completeness, drift, serving SLOs, and outcomes.

Scenario-Based Follow-up Questions

Practise explaining these trade-offs aloud.

Scenario 1: Fraud features are suddenly stale. What do you do?

Check source event rate, stream lag, processing failures, state updates, and online-store health. Quantify affected entities and duration, alert the model owner, and apply the agreed fallback. Recover from durable events where possible, verify freshness and parity, then document root cause.

Scenario 2: Offline AUC is excellent, but production quality drops. How do you investigate?

Compare offline/online feature values, inspect point-in-time joins and label windows, look for leakage, check missing/default rates and freshness, and verify model/feature versions. Then investigate population changes, delayed labels, thresholds, and serving latency.

Scenario 3: You need six months of historical backfill. How do you avoid disruption?

Publish a new feature version, backfill into isolated storage, validate representative periods and point-in-time semantics, coordinate with live writes, then switch consumers only after validation. Retain the previous version for rollback.

Scenario 4: A new 30-day rolling feature is required by a low-latency API. What do you consider?

Estimate event volume, state size, update frequency, latency, and retention. Choose incremental stream aggregates or precomputed values, and define late-event corrections, TTL, recovery, backfill, and offline-training consistency.

Chapter Summary

Strong feature engineering systems are versioned, time-aware data products with explicit freshness guarantees, reliable online serving, historical reproducibility, and monitoring. Connect every design choice to model correctness, inference latency, operational resilience, and business risk.

Practice challenge: Design a fraud feature pipeline that serves predictions within 50 ms, computes 5-minute and 24-hour account aggregates, supports point-in-time training, and backfills a changed feature without interrupting production. Explain storage, late-event handling, monitoring, and rollback.