Senior Data Engineer - Streaming

Experteer Overview In this role you will design and codify enterprise streaming patterns to shift Coca-Cola’s data estate toward a continuous, event-driven model. You’ll build reusable frameworks and templates that enable teams to deploy real-time data streams on Microsoft Fabric, with a focus on Kappa patterns and end-to-end reliability. You will act as the internal authority on streaming design, driving scalable ingestion and robust backfill, with dashboards to monitor health metrics. This is a high-impact opportunity to shape how real-time data supports business SLAs and decision-making. Compensation / Benefits * Build a reusable streaming architecture playbook emphasizing Kappa patterns and near-real-time analytics on Fabric * Architect, document, and templatize replay/backfill mechanisms (offset reset, checkpoint/state management, dedupe strategies) * Develop architectural blueprints and guardrails for sub-second/near-real-time latency aligned with business SLAs * Package advanced streaming concepts (stateful processing, watermarking, late-arriving data remediation, complex windowing) into reusable templates * Codify end-to-end idempotency with deterministic replay and effective-once delivery * Create metadata-driven frameworks that tolerate upstream schema drift without disrupting downstream streaming queries * Serve as internal authority on streaming design across Kafka/Event Hubs for scalable Fabric ingestion * Develop standardized dashboards to monitor end-to-end data lag, throughput, backpressure, and DLQ routing Tasks * 8+ years in data platform architecture * 3+ years architecting/operating production-grade Streaming and Kappa architectures * 4+ years Spark Streaming and Apache Kafka pipelines (topic partitioning, consumer groups, schema registries, high-throughput tuning) * Deep familiarity with Spark Structured Streaming, Kafka Streams, or Apache Flink * Strong Scala and/or PySpark experience with stateful Structured Streaming * Advanced knowledge of Delta Lake append-only sinks and checkpointing * Framework mindset with developer-facing frameworks or configurable streaming engines * Ability to articulate complex streaming concepts to developers and executives Key requirements * medical benefits * financial benefits * annual incentive/support * growth culture * inclusive and agile environment * remote/hybrid options (implied by large company practices)

and streaming concepts (stateful processing, watermarking, late-arriving data remediation, complex windowing) into reusable templates * Codify end-to-end idempotency with deterministic replay and effective-once delivery * Create metadata-driven frameworks that tolerate upstream schema drift without disrupting downstream streaming queries * Serve as internal authority on streaming design across Kafka/Event Hubs for scalable Fabric ingestion * Develop standardized dashboards to monitor end-to-end data lag, throughput, backpressure, and DLQ routing Tasks * 8+ years in data platform architecture * 3+ years architecting/operating production-grade Streaming and Kappa architectures * 4+ years Spark Streaming and Apache Kafka pipelines (topic partitioning, consumer groups, schema registries, high-throughput tuning) * Deep familiarity with Spark Structured Streaming, Kafka Streams, or Apache Flink * Strong Scala and/or PySpark experience with stateful Structured Streaming * Advanced knowledge of aa and Lake append-only sinks and checkpointing * Framework mindset with developer-facing frameworks or configurable streaming engines * Ability to articulate complex streaming concepts to developers and executives Key requirements * medical benefits * financial benefits * annual incentive/support * growth culture * inclusive and agile environment * remote/hybrid options (implied by large company practices)