Data Engineering Manager, Azure Databricks

Nextech Solutions
Pittsburgh, PA, United States
3 days ago
Apply on www.dice.com
Prepare application

Role details

Contract type
Permanent contract
Employment type
Full-time (> 32 hours)
Experience level
Experienced
Experience required
2 years minimum
Working hours
Regular working hours
Job source

Tech stack

Unity 3d Data Analysis Batch Processing Software Quality Code Review Data as a Services Data Architecture Data Validation Information Engineering Data Governance Data Integration Data Warehousing
+16 more
Python (Programming Language) Performance Tuning Software Architecture DataOps Azure Data Lake SQL Databases Data Streaming Data Processing Data Lakes Pyspark Performance Monitor Real Time Data Apache Kafka Data Pipelines Confluent Databricks

Job description

Lead and develop a team of data engineering professionals while overseeing the design, development, and ongoing optimization of the Azure Databricks platform and enterprise data warehouse, supporting business analytics, reporting, and the evolution toward real-time, event-driven data integration., ? Own source-system ingestion, Delta Lake pipelines, Medallion Architecture, and enterprise data warehouse design, development, and operations within Azure Databricks. ? Provide hands-on technical leadership across Azure Databricks, Azure Data Lake Storage, Unity Catalog, PySpark, Python, and SQL including architecture reviews, code reviews, performance optimization, and technical mentoring. ? Support transition from batch processing to continuous, near-real-time data ingestion and processing using Azure Event Hubs and Structured Streaming to meet demanding service-level requirements within a highly regulated environment. ? Monitor and optimize data warehouse reliability, processing performance, scalability, and cost efficiency while maintaining the integrity and availability of critical business data. ? Direct production incident response, troubleshoot complex pipeline and data processing issues, and establish proactive monitoring, documentation, and knowledge-sharing practices. ? Establish and enforce engineering standards for code quality, testing, deployment, security, and documentation, ensuring reliable, governed data operations within a highly regulated healthcare environment. ? Collaborate with Data Architecture, Analytics Engineering, QA, and Data Governance on Gold-layer design, data validation, lineage, metadata, and observability. ? Support internal business requirements and external partner data services through the development, enhancement, and integration of data pipelines. ? Lead, coach, mentor, and develop a team of Data Engineers, establishing clear accountability, managing performance, and fostering a collaborative, high-performing engineering culture prepared for significant growth.

Requirements

? 7+ years of progressive data engineering experience, including 2+ years of team leadership or people management. ? Deep, hands-on Databricks expertise, including medallion architecture, Delta Lake, Unity Catalog, and production pipeline development and operations. ? Strong enterprise data warehouse experience encompassing architecture, data modeling, transformation logic, and production operations. ? Proficiency in PySpark, SQL, and Python with the technical depth to review code, evaluate architectural decisions, and mentor engineers. ? Proven experience with Databricks, Azure Data Lake Storage, and Azure Event Hubs. ? Strong experience developing and operating data pipelines, preferably with Azure Event Hubs or comparable technologies such as Kafka, Kinesis, or Confluent Cloud. ? Demonstrated ability to lead and develop engineers, manage competing priorities, and maintain accountability in a high-volume production environment. ? Strong communication, organizational, and cross-functional collaboration skills with the ability to work effectively across technical, business, partner, and client teams.

Apply for this position

This job is hosted externally. Click below to view the full posting and apply.

Apply on www.dice.com
Prepare application

Good distractions

Loading talks and stories from around this role…