Lead DATA ENGINEER - Cloud Migration
- Discuss this with your agent
- Open in Claude
- Open in ChatGPT
Prepare application
- Draft this with your agent
- Open in Claude
- Open in ChatGPT
Role details
Tech stack
+11 more
Job description
This role is part of a multiyear enterprise initiative to modernize data platforms by migrating from legacy and onprem environments to cloudnative, governed, and scalable architectures . The role focuses on migrating enterprise data workloads to Databricks on strategic cloud platforms , enabling standardized data engineering, analytics, centralized reporting, reconciliation utilities, and AI/ML use caseswhile adhering to controls, security, resilience, and regulatory compliance ., * Lead and execute migration of legacy data platforms (onprem / nonstandard tools) to Databricks on cloud under the Olympus program
- Perform application, data, and pipeline refactoring to cloudnative Databricks patterns
- Drive migration planning including dependency analysis, sequencing, and cutover strategy
- Support coexistence models and transition from dualrun to cloudonly execution
Databricks Lakehouse Engineering
- Design and implement Databricks Lakehouse architecture (Bronze / Silver / Gold)
- Build scalable batch and streaming pipelines using PySpark, Spark SQL
- Leverage Delta Lake for reliability, versioning, and performance
- Optimize compute usage and cost in line with enterprise cloud efficiency goals
Enterprise Data Controls & Governance
- Embed data quality, reconciliation, and completeness controls as part of migration
- Ensure migrated workloads meet EDO governance, MCA, and audit requirements
- Maintain lineage, traceability, and explainability across migrated assets
- Support riskcritical use cases (Finance, Ops, Recon, Reporting)
Cloud Security & Resilience
- Implement cloudaligned RBAC, identity controls, and secure access patterns
- Enforce data encryption, masking, and classification standards
- Ensure workloads meet operational resilience and recovery expectations
- Partner with cloud platform and security teams for certification and signoff
Reporting, Analytics & AI Enablement
- Enable downstream BI, regulatory reporting, and MI workloads on Databricks
- Support centralized reporting programs (e.g., ARA, GRUrelated use cases)
- Prepare data foundations for AI / ML and Agentic workflows postmigration
Requirements
Software Engineering, Cloud Computing, Reconciliation, Regulatory Compliance, Software Engineering, Cloud Computing, Artificial Intelligence (AI), Data Migration, Data Analysis, Refactoring, * 812+ years in data engineering / platform modernization
- Strong handson experience with Databricks in largescale enterprises
- Proven experience delivering cloud migration programs (onprem cloud)
- Deep expertise in Apache Spark, PySpark, Spark SQL
- Experience embedding controls, reconciliation, and data quality in migrations
- Experience in regulated environments (banking / financial services preferred), * Experience with Client Olympus or equivalent enterprise cloud programs
- Knowledge of legacy data platforms and modernization patterns
- Familiarity with Finance, Ops, Recon, or BalanceSheet data domains
- Exposure to MLflow, AI pipelines, or GenAI enablement on cloud
- Strong understanding of runthebank vs changethebank execution
Behavioral & Delivery Expectations
- Strong ownership and execution mindset
- Comfortable operating in large, multivendor transformation programs
- Ability to engage with Technology, Operations, Risk, and Audit stakeholders
- Disciplined approach to migration risk, controls, and documentation
Apply for this position
This job is hosted externally. Click below to view the full posting and apply.
Prepare application
- Draft this with your agent
- Open in Claude
- Open in ChatGPT
Good distractions
Loading talks and stories from around this roleβ¦