> Markdown version of [/jobs/ext/4716873-senior-data-engineer-azure-databricks](https://www.wearedevelopers.com/jobs/ext/4716873-senior-data-engineer-azure-databricks). Every page supports `.md` or `Accept: text/markdown`. Links point to the HTML versions so they work for humans too. Agent guide: [/agents.md](https://www.wearedevelopers.com/agents.md). --- # Senior Data Engineer, Azure Databricks - **Company:** Nextech Solutions - **Location:** Pittsburgh, PA, United States (Remote available) - **Experience:** Expert - **Contract:** Permanent contract - **Skills:** Data Analysis, Automation of Tests, Microsoft Azure, Batch Processing, Software Quality, Code Review, Continuous Integration, Data Architecture, Data Validation, Information Engineering, Data Governance, Data Infrastructure, Data Integration, Data Integrity, Data Security, Data Warehousing, Digital Assets, Dimensional Modeling, Python (Programming Language), Metadata, Meta-Data Management, Operational Data Store, Operational Databases, Performance Tuning, DataOps, Azure Data Lake, SQL Databases, Data Streaming, Enterprise Data Management, Data Processing, Data Ingestion, Azure Data Factory, Apache Spark, Data Lakes, Pyspark, Collibra, Low Latency, Apache Kafka, Software Version Control, Data Pipelines, Confluent, Databricks - **Published:** November 14, 2026 - **Apply:** https://www.dice.com/job-detail/62cb21e4-16bd-44c8-8004-5574af3d098e ## About the Role 6+ years of progressive data engineering experience including hands-on development and production support of enterprise data platforms. Deep expertise in Azure Databricks, Delta Lake, Medallion Architecture, Unity Catalog, and production pipeline development and optimization. Hands-on experience developing and operating event-driven streaming data pipelines using Azure Event Hubs, Kafka, AWS Kinesis, or Confluent Cloud. Strong enterprise data warehousing experience encompassing dimensional modeling, complex transformations, data integration, and production operations. Advanced proficiency in SQL, Python, and PySpark, with the ability to independently develop, troubleshoot, review, and optimize complex data pipelines. Hands-on experience with Azure Data Lake Storage, Azure Data Factory, and CI/CD practices using Azure DevOps. Demonstrated ownership of production data pipelines, including monitoring, incident response, performance optimization, and service-level accountability. Experience implementing data contracts using ODCS or equivalent standards preferred. Experience implementing data validation, reconciliation, automated testing, and data quality controls. Strong understanding of data governance, metadata management, lineage, and secure data handling practices. Familiarity with data observability platforms such as Monte Carlo preferred. Experience operating within highly regulated environments including healthcare, pharmacy, pharmaceutical, medical, or another highly regulated industry involving sensitive clinical or operational data is preferred. Ability to analyze complex legacy data environments, improve existing architectures, and produce comprehensive technical documentation. Familiarity with Structured Streaming, Zerobus Ingestion, and Spark Declarative Pipelines preferred. Experience with enterprise data catalog platforms such as Atlan, Collibra, or equivalent technologies preferred. Experience developing governed data assets with documented metadata, lineage, and access controls. Strong communication, problem-solving, and cross-functional collaboration skills. ## Description The Senior Data Engineer will design, develop, and optimize enterprise data warehouse solutions and production data pipelines within an Azure Databricks environment. This hands-on position will support the continued development and modernization of the data ecosystem including the transition toward real-time, event-driven data processing, while ensuring data quality, reliability, and performance in a highly regulated environment., Design, develop, and maintain enterprise data warehouse solutions and production pipelines across Bronze, Silver, and Gold layers using Azure Databricks, Delta Lake, and medallion architecture. Develop and optimize data ingestion, transformation, and modeling solutions using SQL, Python, PySpark, and Azure Data Factory. Build dimensional and harmonized data models, incorporating data conformance, survivorship logic, and standardized transformation processes. Support the transition from traditional batch processing toward continuous, low-latency, event-driven data ingestion using Azure Event Hubs, Structured Streaming, and emerging Databricks capabilities. Own assigned production data domains, ensuring pipeline reliability, data integrity, performance, and compliance with established service-level requirements. Monitor and optimize pipeline performance, processing latency, scalability, and Databricks compute costs. Troubleshoot complex production issues, perform root-cause analysis, and implement sustainable improvements to data reliability and system performance. Implement automated data validation, reconciliation, quality controls, and testing practices throughout the data lifecycle. Support data governance initiatives involving Unity Catalog, metadata management, lineage, access controls, and Gold-layer data assets. Modernize and consolidate legacy transformation processes while maintaining data accuracy and operational continuity. Establish and maintain engineering best practices for code quality, testing, documentation, version control, and CI/CD deployment within a highly regulated environment. Collaborate with Data Architecture, Analytics Engineering, QA, Data Governance, and business stakeholders to deliver reliable, scalable data solutions. Develop and enhance internal and external partner data integrations to support evolving business and operational requirements. Provide technical guidance and mentorship to Data Engineers through code reviews, knowledge sharing, and engineering best practices.