Senior Data Engineer, Azure Databricks
- Discuss this with your agent
- Open in Claude
- Open in ChatGPT
Prepare application
- Draft this with your agent
- Open in Claude
- Open in ChatGPT
Role details
Tech stack
+30 more
Job description
The Senior Data Engineer will design, develop, and optimize enterprise data warehouse solutions and production data pipelines within an Azure Databricks environment. This hands-on position will support the continued development and modernization of the data ecosystem including the transition toward real-time, event-driven data processing, while ensuring data quality, reliability, and performance in a highly regulated environment., Design, develop, and maintain enterprise data warehouse solutions and production pipelines across Bronze, Silver, and Gold layers using Azure Databricks, Delta Lake, and medallion architecture. Develop and optimize data ingestion, transformation, and modeling solutions using SQL, Python, PySpark, and Azure Data Factory. Build dimensional and harmonized data models, incorporating data conformance, survivorship logic, and standardized transformation processes. Support the transition from traditional batch processing toward continuous, low-latency, event-driven data ingestion using Azure Event Hubs, Structured Streaming, and emerging Databricks capabilities. Own assigned production data domains, ensuring pipeline reliability, data integrity, performance, and compliance with established service-level requirements. Monitor and optimize pipeline performance, processing latency, scalability, and Databricks compute costs. Troubleshoot complex production issues, perform root-cause analysis, and implement sustainable improvements to data reliability and system performance. Implement automated data validation, reconciliation, quality controls, and testing practices throughout the data lifecycle. Support data governance initiatives involving Unity Catalog, metadata management, lineage, access controls, and Gold-layer data assets. Modernize and consolidate legacy transformation processes while maintaining data accuracy and operational continuity. Establish and maintain engineering best practices for code quality, testing, documentation, version control, and CI/CD deployment within a highly regulated environment. Collaborate with Data Architecture, Analytics Engineering, QA, Data Governance, and business stakeholders to deliver reliable, scalable data solutions. Develop and enhance internal and external partner data integrations to support evolving business and operational requirements. Provide technical guidance and mentorship to Data Engineers through code reviews, knowledge sharing, and engineering best practices.
Requirements
6+ years of progressive data engineering experience including hands-on development and production support of enterprise data platforms. Deep expertise in Azure Databricks, Delta Lake, Medallion Architecture, Unity Catalog, and production pipeline development and optimization. Hands-on experience developing and operating event-driven streaming data pipelines using Azure Event Hubs, Kafka, AWS Kinesis, or Confluent Cloud. Strong enterprise data warehousing experience encompassing dimensional modeling, complex transformations, data integration, and production operations. Advanced proficiency in SQL, Python, and PySpark, with the ability to independently develop, troubleshoot, review, and optimize complex data pipelines. Hands-on experience with Azure Data Lake Storage, Azure Data Factory, and CI/CD practices using Azure DevOps. Demonstrated ownership of production data pipelines, including monitoring, incident response, performance optimization, and service-level accountability. Experience implementing data contracts using ODCS or equivalent standards preferred. Experience implementing data validation, reconciliation, automated testing, and data quality controls. Strong understanding of data governance, metadata management, lineage, and secure data handling practices. Familiarity with data observability platforms such as Monte Carlo preferred. Experience operating within highly regulated environments including healthcare, pharmacy, pharmaceutical, medical, or another highly regulated industry involving sensitive clinical or operational data is preferred. Ability to analyze complex legacy data environments, improve existing architectures, and produce comprehensive technical documentation. Familiarity with Structured Streaming, Zerobus Ingestion, and Spark Declarative Pipelines preferred. Experience with enterprise data catalog platforms such as Atlan, Collibra, or equivalent technologies preferred. Experience developing governed data assets with documented metadata, lineage, and access controls. Strong communication, problem-solving, and cross-functional collaboration skills.
Apply for this position
This job is hosted externally. Click below to view the full posting and apply.
Prepare application
- Draft this with your agent
- Open in Claude
- Open in ChatGPT
Good distractions
Loading talks and stories from around this roleβ¦