Senior Data Engineer

This role will involve designing and maintaining robust data infrastructure, ensuring data quality, and implementing best practices in DevOps and CI/CD pipelines. The ideal candidate will have extensive experience in cloud data engineering, particularly with Databricks and Snowflake, and a strong understanding of healthcare data. WHAT YOU’LL DO: Lead the migration of SQL Server to Databricks and Snowflake, including schema conversion, ETL re-platforming, data validation, and cutover planning. Design and maintain Databricks infrastructure, focusing on Unity Catalog, Bronze/Silver/Gold medallion architecture, cluster/job orchestration, and cost optimization. Build and manage Snowflake environments, including warehouses, RBAC, data sharing, and performance tuning. Own CI/CD pipelines for data engineering workflows using Azure DevOps, GitHub Actions, or similar tools, ensuring automated testing, deployment, and rollback. Implement Infrastructure as Code (IaC) using Terraform or Bicep for repeatable and auditable environment provisioning. Establish DevOps best practices, including branching strategy, code review gates, and environment promotion (dev/test/prod). Build monitoring, alerting, and observability for pipeline health and data quality. Collaborate with data engineers, analysts, and data scientists across various scrum teams to ensure infrastructure supports downstream needs, including HEDIS, quality, and medical economics. Ensure HIPAA-compliant security posture across cloud environments, focusing on encryption, access controls, and audit logging

We are seeking a highly skilled Senior Data Engineer to lead the migration of our SQL Server databases to Databricks and Snowflake., 5+ years of experience in cloud data engineering/DevOps, with hands-on experience in Databricks and Snowflake in production. Strong design experience with CI/CD pipelines (Azure DevOps, GitHub Actions, Jenkins, or GitLab CI). Proven experience in SQL Server-to-cloud migration. Proficiency in Infrastructure as Code (Terraform, Bicep, or CloudFormation). Fluency in Python and SQL; experience with PySpark is strongly preferred. Solid understanding of medallion architecture, data lineage, and governance patterns. Experience with Azure cloud platform (preferred) or AWS/GCP. PREFERRED SKILLS: Experience with healthcare/payer data, including HEDIS, claims, and clinical data. Exposure to Databricks Unity Catalog and MLflow. Familiarity with Genie/AI-BI or similar natural-language query tools