Lead Data Engineer

We are seeking a Senior/Lead Data Engineer to modernize and own enterprise data pipelines by migrating legacy SAS-based workflows to Python/PySpark on Databricks. The engineer will partner with SMEs during the transition period and eventually take ownership of a critical healthcare analytics pipeline supporting HEDIS reporting., * Migrate legacy SAS pipelines to Python/PySpark on Databricks.

• Design, build, and maintain scalable ETL/ELT pipelines.
• Develop distributed data processing solutions using Databricks.
• Create and optimize complex SQL queries.
• Schedule, automate, and monitor data pipelines.
• Work with AWS services including S3, Lambda, Glue, and EC2.
• Manage Databricks notebooks, workflows, and clusters.
• Collaborate with business stakeholders and SMEs to transition pipeline ownership.
• Follow Agile methodologies and Git-based development practices.

• 8+ years of Data Engineering experience.
• Strong Python programming and scripting skills.
• Hands-on PySpark development.
• Extensive Databricks experience (Notebooks, Workflows, Cluster Management).
• Experience processing large datasets in distributed environments.
• Strong SQL skills.
• AWS experience with:
• S3
• Glue
• Lambda
• EC2
• Experience building and automating ETL/ELT pipelines.
• Git/version control experience.
• Agile/Scrum experience.

Preferred Skills

• Experience with SAS-to-Python migration.
• AI/Automation experience.
• Healthcare or HEDIS domain experience.
• Lead/Principal Data Engineering experience.

Must-Have Skills

• Python
• PySpark
• Databricks
• SQL
• AWS (S3, Glue, Lambda, EC2)
• ETL/ELT Pipeline Development
• Distributed Data Processing
• Git
• Agile/Scrum

Nice-to-Have Skills

• SAS Modernization
• AI/Generative AI
• Healthcare/HEDIS
• Airflow or other workflow orchestration tools
• Leadership/Pipeline Ownership experience