Principal Data Engineer

CGT
Pittsburgh, PA, United States
2 days ago
Apply on www.dice.com
Prepare application

Role details

Contract type
Permanent contract
Employment type
Full-time (> 32 hours)
Experience level
Experienced
Experience required
2 years minimum
Working hours
Regular working hours
Job source

Tech stack

Airflow Microsoft Azure Continuous Integration Data Architecture Information Engineering Data Governance Data Systems Python (Programming Language) Windows PowerShell SQL Databases Data Streaming Enterprise Data Management
+10 more
Data Processing Azure Data Factory Apache Spark Data Lakes Pyspark Apache Kafka Data Management Machine Learning Operations Data Pipelines Databricks

Job description

We are seeking an experienced Data Engineering Manager to lead and support the development of modern, scalable data platforms. This individual will play a key role in designing, building, and optimizing enterprise-level data solutions using Microsoft Azure and Azure Databricks. The ideal candidate will bring strong hands-on data engineering experience, particularly with Azure Databricks, PySpark, Python, SQL, and Lakehouse Architecture. This role is suited for a technical leader who enjoys remaining involved in development while guiding and mentoring data engineering teams. We are also open to experienced Lead Data Engineers, Principal Data Engineers, and Databricks Technical Leads who have strong technical expertise and leadership capabilities., * Lead the design, development, and implementation of scalable enterprise data platforms using Azure Databricks.

  • Build, maintain, and optimize batch and streaming data pipelines using Azure Data Factory and other Azure data services.
  • Design and implement Lakehouse Architecture, Delta Lake, and Medallion Architecture.
  • Develop and optimize data processing solutions using PySpark, Python, SQL, and Apache Spark.
  • Improve data pipeline performance, scalability, reliability, and efficiency.
  • Implement data governance, security, and quality standards across enterprise data platforms.
  • Provide technical leadership, mentorship, and guidance to data engineering team members.
  • Collaborate with cross-functional teams to deliver reliable data solutions that support business and analytics needs.

Requirements

  • 5-7+ years of experience in data engineering, data architecture, or related technical roles.
  • 2-4 years of hands-on experience working with Azure Databricks.
  • Strong experience with PySpark, Python, SQL, and Apache Spark.
  • Hands-on experience with Delta Lake, Lakehouse Architecture, and Medallion Architecture.
  • Experience building and managing batch and streaming data pipelines using Azure Data Factory and Azure data services.
  • Strong understanding of data governance, pipeline optimization, and Azure cloud infrastructure.
  • Experience designing and implementing production-level data engineering solutions.

Nice-to-Have Qualifications

  • Experience managing, leading, or mentoring data engineering teams.
  • Knowledge of Scala and PowerShell.
  • Experience with Unity Catalog, Kafka, MLflow, and Apache Airflow.
  • Familiarity with Azure DevOps and CI/CD practices.
  • Relevant certifications, including Azure Data Engineer, Databricks Data Engineer, or Azure Solutions Architect.

Apply for this position

This job is hosted externally. Click below to view the full posting and apply.

Apply on www.dice.com
Prepare application

Good distractions

Loading talks and stories from around this role…