System Engineer

Senior backend engineer building the core integrations between Unity and the enterprise Spark engine and SparkFlow framework

• Designs distributed, high-throughput systems that power data processing and integration workflows at enterprise scale

• Operates with significant autonomy, setting technical direction within a regulated, large-scale engineering environment

Key Responsibilities

Platform & Integration Engineering

• Design, build, and operate integrations between Unity and the Spark engine and SparkFlow framework

• Develop robust APIs, SDKs, and connectors enabling reliable data movement and workflow orchestration

• Implement abstractions that simplify SparkFlow adoption for downstream platform consumers

Distributed Systems Design

• Architect scalable, fault-tolerant systems handling large data volumes and concurrent workloads

• Apply best practices for performance tuning, caching, partitioning, backpressure, and resilience

• Drive design reviews, RFCs, and architectural decisions for core platform components

Reliability & Operations

• Build systems with first-class observability (metrics, logs, traces) and SLOs

• Establish operational excellence: runbooks, alerting, capacity planning, and incident response

• Champion automated testing, and progressive delivery practices

6 years of backend engineering experience, with deep expertise in distributed systems

• Expert-level Python; strong API design (REST, gRPC) and asynchronous programming

• Proven track record delivering large-scale, production-grade systems

• Strong fundamentals in concurrency, data structures, and systems performance experience

Preferred Qualifications

• Hands-on experience with Apache Spark (PySpark, Spark SQL, Structured Streaming)

• Experience building or integrating with workflow/data orchestration frameworks (Airflow, Autosys)

• Exposure to greenfield platform builds in regulated financial enterprises

• Experience with event-driven architectures (Kafka) and lakehouse formats (Iceberg, Delta, Hudi)

Technical Skills

• Languages: Python (expert), Scala or Java (plus), SQL

• Distributed Compute: Apache Spark

• Storage & Data: S3/ADLS/GCS, Iceberg, Delta Lake, Parquet, Postgres

• Streaming & Messaging: Kafka, Pub/Sub

• APIs & Services: REST, gRPC, GraphQL, OpenAPI

• Containers & Cloud: Docker, Kubernetes, Azure/GCP, Terraform

• CI/CD: GitHub Actions

Soft Skills & Leadership Expectations

• Strong technical judgment and ability to operate