Sr. Data Engineer

We are seeking a highly skilled Senior Data Engineer to join our Data Engineering & Platform organization. This role is ideal for someone passionate about building scalable data solutions, enabling cross-functional data delivery, and raising engineering excellence across the organization. You will work on an advanced Databricks-on-AWS data platform and play a key role in transforming our data ecosystem-designing and building resilient pipelines, implementing CI/CD, and contributing to platform modernization efforts.

As a senior member of the team, you will partner with data engineers, analytics teams, product teams, and platform engineering to build high-quality, production-grade data solutions that power enterprise analytics, data science, and operational workloads., Data Engineering & Pipeline Development

• Design, build, maintain, and optimize scalable data pipelines and ELT/ETL processes to ingest, process, and store large volumes of data from various sources.
• Architect robust batch and streaming solutions using PySpark, Spark SQL, and Databricks Jobs.
• Monitor and troubleshoot data pipelines and infrastructure to ensure high availability and performance.
• Ensure reliability, observability, and SLAs through monitoring, alerting, and automated recovery patterns.
• Implement and maintain data integration solutions, including APIs and data connectors, to facilitate data exchange between systems.

Cloud Platform Engineering

• Build and manage data solutions on AWS, leveraging services such as S3, Lambda, Glue, IAM, EC2, EKS, Step Functions, and EventBridge.
• Apply cloud engineering best practices for cost optimization, reliability, and security.

CI/CD, DevOps, and Infrastructure-as-Code

• Implement and maintain CI/CD pipelines using:

• Databricks Asset Bundles
• Terraform (IaC)
• GitHub Actions / Azure DevOps / Jenkins (depending on internal stack)

• Own deployment automation and environment management for data workloads and platform components.
• Contribute to reusable Terraform modules and engineering standards.

Data Governance and Data Quality

• Maintain data quality through validation frameworks (e.g., DQX, Great Expectations, Databricks expectations, custom frameworks).
• Ensure data quality, integrity, and security by implementing best practices for data governance and management.
• Implement governance and lineage through Unity Catalog, cataloging standards, and metadata best practices.

Platform Modernization

• Contribute to initiatives such as:

• Migration of pipelines from legacy integration tools to Lakeflow / Lakehouse-native ingestion
• Improving infrastructure automation
• Enhancing data observability, testing, and deployment workflows

Collaboration & Technical Leadership

• Serve as a technical mentor to junior engineers.
• Partner closely with data analysts, data scientists, and business stakeholders to deliver enterprise data assets.
• Contribute to architecture decisions, design reviews, and platform roadmap conversations.

Do you have experience in Terraform?, * Must be authorized to work in the U.S. without the need for employment-based visa sponsorship now or in the future. We are unable to sponsor applicants for U.S. work visa status for this opportunity (no sponsorship is available for H-1B, L-1, O-1, E-3, H-1B1, F-1, J-1, OPT, CPT or any other employment-based visa)., * 7+ years of data engineering experience building production-grade data pipelines.

• Deep expertise in Databricks , Spark , and Delta Lake .
• Hands-on experience with AWS cloud services.
• Strong proficiency in Python , SQL , and distributed computing patterns.
• Experience with Terraform and cloud-native CI/CD.
• Strong understanding of data modeling, orchestration, and data quality frameworks.
• Ability to operate autonomously while collaborating in a team environment.

Additional Qualifications

• Experience with Lakeflow, Databricks Workflows, or ingestion modernization initiatives.
• Knowledge of software engineering best practices:

• test automation
• versioning
• code reviews
• Git workflows

• Familiarity with event-driven architecture and streaming technologies.
• Experience supporting Data Science and ML platform teams.
• P&C Insurance domain knowledge preferred