Databricks Platform Engineer

• Design and build batch data pipelines using Apache Spark on EKS and Databricks, processing data from NAS and upstream source systems into Iceberg tables on S3
• Own the Bronze Silver Gold DLT pipeline architecture for assigned business domains
• Write and maintain DDL for Iceberg tables following the CDP naming standard (catalog.schema.table with region, LOB, and layer components)
• Integrate with existing loading framework (Autosys OpenShift S3 Airflow DAG trigger chain) and extend it as the platform evolves
• Work with Data Stewards and DBA team to ensure schema design meets data classification requirements (INT, PII, CPII, CSPII)
• Implement and validate Unity Catalog governance controls - column masking, row-level filters, and lineage tracking - for tables under your ownership

• Contribute to the migration of pipeline artifacts from DBFS to UC Volumes as part of the platform’s governance uplift programmer DataOps Responsibilities

• Own the operational health of pipelines in your domain - monitoring, alerting, failure triage, and restart procedures
• Build and maintain job monitoring integration between Databricks / Airflow and the team’s observability stack (Splunk, CloudWatch, MSK audit events)
• Define and document incident runbooks for common failure patterns: EKS resource exhaustion, S3 VPC Endpoint connectivity, Spark executor out-of-memory, DLT pipeline schema evolution failures
• Manage deployment of pipeline code and DAG updates through the Lightspeed CI/CD pipeline - understand what goes to S3 config bucket, what goes to the DAG bucket, and what goes to EFS
• Participate in on-call rotation for production pipeline incidents
• Work with the platform team on capacity planning for EKS cluster - there is no YARN queue here, resource exhaustion means job failure

• Maintain data quality monitoring and reconciliation between on-prem and cloud outputs during the migration period What We Are Looking For, * Building dashboards or BI reports

• Managing the Databricks platform itself - that sits with the platform team The Environment

• Compute: EKS (Spark), OpenShift (on-prem loading framework), Databricks (UC, MLflow, DLT)
• Storage: S3 (Iceberg), EFS (intermediate artifacts), NAS (source files)
• Orchestration: Airflow (MWA), Autosys (on-prem, being phased out)
• Messaging: MSK (Kafka), SQS, SNS
• Catalog: HMS on Aurora migrating to AWS Glue Unity Catalog
• Security: Vault (credential injection), KMS (encryption), PrivateLink (no public internet for data paths), CloudTrail (audit)

• CI/CD: Lightspeed (Citi internal), Bitbucket / GitHub What Good Looks Like in This Role In the first 90 days, you should be able to:

• Explain the end-to-end data flow from NAS to an Iceberg table in S3, naming every component the data touches and the network path it takes
• Run a Spark job on EKS independently, debug a failure from the pod logs, and know when to look at the Spark History Server vs CloudTrail
• Identify a governance gap in a UC table configuration and propose a fix In six months, you should own at least one business domain end-to-end - pipeline design, operational health, schema governance, and incident response.

Must Have

• Overall 10 years’ experience; 4+ years of data engineering experience, with at least 2 years working with Spark in a production environment
• Solid understanding of cloud networking fundamentals - VPC, subnets, VPC Endpoints, Security Groups. You do not need to be a network engineer, but you need to understand why a Spark pod’s network path matters and what breaks when it is wrong
• Hands-on experience with AWS services: S3, EKS, IAM, KMS, SQS, MSK
• Experience with Apache Iceberg or Delta Lake in production - not just awareness, but actual table design and maintenance
• Familiarity with Kubernetes - you do not need to administer a cluster, but you need to understand pod l ifecycle, resource limits, and how to read a pod describe output when something fails
• Comfortable reading and writing Python and SQL at a production standard

• Understanding of IAM and credential management - why static keys are unacceptable, how STS AssumeRole works, what Vault does Strong Preference

• Experience with Databricks, specifically Unity Catalog, MLflow, and DLT
• Experience working in a regulated financial services environment - you understand why data classification, audit trails, and access controls are not optional extras
• Familiarity with Airflow (DAG design, KubernetesPodOperator, trigger chain design)
• Experience with Spark performance tuning - you know what data skew looks like, when to add executors vs CPU, and how to read a Spark history server

• Exposure to model governance or MLOps - understanding how MLflow Model Registry, Champion/Challenger alias management, and MRM review processes fit together Not Required But Noticed

• Experience with Lakebase or other OLTP-on-lakehouse technologies
• Familiarity with Collibra or similar enterprise data governance platforms
• Experience designing UC group naming conventions or RBAC frameworks at scale, BACHELOR OF COMPUTER SCIENCE