AWS Databricks Data Engineer

• Design and implement batch and streaming data pipelines using PySpark, Delta Lake, and Delta Live Tables (DLT).
• Develop scalable ingestion and transformation solutions within Databricks.
• Build integrations with AWS services such as S3, IAM, and Lambda.
• Configure and manage data governance, security, and access controls via Databricks Unity Catalog.
• Monitor and optimize pipeline performance, reliability, and data quality.
• Automate testing and deployment using CI/CD tools (GitHub/GitLab).
• Collaborate with data scientists and stakeholders to prepare datasets for analytics and machine learning.
• Support healthcare payer datasets including claims, provider, member enrollment, and MMIS data.

We are seeking an experienced AWS Databricks Data Engineer to design, build, and maintain scalable data pipelines and cloudnative data infrastructure. The ideal candidate will have strong expertise in Apache Spark, PySpark, Python, SQL, Delta Lake, and AWS services, with proven experience in production Databricks environments., * 3 5 years of handson data engineering experience.

• Strong productionlevel experience with Databricks.
• Advanced programming skills in Python and PySpark.
• Advanced SQL expertise.
• Deep understanding of Apache Spark, Delta Lake, and Data Lakehouse architecture.
• Experience with Delta Live Tables, Unity Catalog, and Databricks Workflows.
• Familiarity with AWS cloud services for data storage and management.