Data Engineer Python / PySpark / Databricks

• Design and build new tools and engines using Python, PySpark, and Databricks.
• Maintain, upgrade, and migrate existing engines, including legacy stored procedures.
• Develop and enhance data pipelines and automation workflows.
• Analyze and process source data and modify or transform engine logic as required.
• Support quality measure and rule development, including clinical measures.
• Collaborate with onshore and offshore teams., The Databricks Architect will be responsible for designing, implementing, and governing enterprise-scale data platforms on Databricks. The role focuses on data architecture, cloud integration, data engineering best practices, analytics enablement, and platform optimization across Azure, AWS, or Google Cloud Platform., * Design and implement scalable data lakehouse architectures using Databricks.
• Define enterprise data models, governance standards, and security controls.
• Lead migration of legacy data warehouses and ETL workloads to Databricks.
• Architect batch and real-time data processing solutions using Spark.
• Optimize performance, scalability, reliability, and cost efficiency.
• Collaborate with business stakeholders, data engineers, data scientists, and cloud architects.
• Establish CI/CD, monitoring, and operational best practices.
• Provide technical leadership and mentor engineering teams.

• Strong hands-on programming experience and ability to build solutions independently.
• Deep experience with Python and PySpark Mandatory.
• Working experience with Databricks.
• Strong experience with stored procedures and legacy-to-modern migration.
• Strong understanding of data and ability to interpret and transform data.
• Healthcare data, clinical quality measures, or rule development experience strongly preferred., * 8+ years of data engineering and data platform experience.
• 3+ years of hands-on Databricks architecture experience.
• Strong expertise in Apache Spark, PySpark, SQL, and Delta Lake.
• Experience with Azure Databricks, AWS Databricks, or Google Cloud Platform Databricks.
• Strong expertise in Google Cloud Platform Pub/Sub, Azure Event Hubs, AWS Kinesis, or Apache Kafka.
• Experience with producer/consumer architecture decisions and trade-offs.
• Knowledge of data lakehouse architecture and data governance.
• Experience with data integration tools and orchestration frameworks.
• Strong understanding of security, access management, and compliance requirements.

Preferred Skills

• Databricks Certified Data Engineer Professional or equivalent.
• Experience with Unity Catalog, Delta Live Tables, and Databricks Workflows.
• Experience with Power BI, Tableau, or other BI platforms.
• Knowledge of DataOps, DevOps, and MLOps practices.

Education

• Bachelor’s degree in Computer Science, Engineering, Information Systems, or related field.
• Master’s degree preferred.