Manager Data Platforms

CGT
Pittsburgh, PA, United States
1 day ago
Apply on www.dice.com
Prepare application

Role details

Contract type
Permanent contract
Employment type
Full-time (> 32 hours)
Experience level
Experienced
Experience required
2 years minimum
Working hours
Regular working hours
Job source

Tech stack

Unity 3d Application Programming Interfaces (APIs) Data Analysis Microsoft Azure Cloud Computing Cloud Computing Security Code Review Computer Programming Databases Data Architecture Information Engineering Data Governance
+30 more
Data Infrastructure Extract Transform Load (ETL) Relational Databases Distributed Computing Environment Identity and Access Management Python (Programming Language) Machine Learning SQL Azure NoSQL Windows PowerShell Standard Sql Azure Data Lake Scala (Programming Language) SQL Databases Data Streaming Enterprise Data Management Data Ingestion Azure Data Factory Apache Spark Data Lakes Pyspark Information Technology Data Lineage Deployment Automation Apache Kafka Data Management Azure Synapse Analytics Data Pipelines Key Vault Databricks

Job description

Seeking an experienced Data Platform Manager to lead the design, development, and optimization of enterprise data platforms within Microsoft Azure. This position will oversee Azure Databricks architecture, data engineering, ETL/ELT pipelines, data governance, and cloud infrastructure integration. The ideal candidate combines hands-on expertise in Databricks, PySpark, SQL, and Azure data services with technical leadership experience managing data engineering teams and delivering scalable analytics solutions., * Lead, mentor, and provide technical direction to data engineers, including code reviews, development standards, and production troubleshooting.

  • Design, build, and maintain scalable enterprise data platforms using Azure Databricks and Microsoft Azure.
  • Architect and implement Lakehouse solutions using Delta Lake and Medallion Architecture (Bronze, Silver, and Gold layers).
  • Oversee the development of batch and real-time ETL/ELT pipelines using PySpark, Scala, SQL, and Apache Spark.
  • Integrate data from APIs, relational databases, streaming platforms, and other enterprise sources.
  • Develop and optimize data ingestion, transformation, enrichment, and validation processes.
  • Integrate Azure Databricks with Azure Data Factory, Azure Data Lake Storage Gen2, Azure SQL Database, and Azure Synapse.
  • Implement Unity Catalog for data governance, access controls, data lineage, privacy, and security.
  • Optimize Databricks clusters, Spark jobs, query execution, and cloud resource utilization for performance and cost efficiency.
  • Support streaming data integrations using Apache Kafka and other distributed processing technologies.
  • Implement CI/CD pipelines and deployment automation using Azure DevOps.
  • Collaborate with data scientists and analysts to support machine learning models, analytics, and business intelligence initiatives.
  • Establish data engineering standards, quality controls, monitoring, and operational best practices.
  • Partner with business stakeholders to translate data requirements into scalable technical solutions.
  • Support cloud infrastructure security, networking, identity management, and access controls.

Requirements

  • Bachelor’s degree in Computer Science, Engineering, or a related field.
  • Five to seven or more years of hands-on experience in data engineering or data architecture.
  • Two to four years of direct experience with Azure Databricks and Azure cloud technologies.
  • Strong hands-on experience designing and implementing Azure Databricks Lakehouse architectures.
  • Advanced programming experience with Python, PySpark, SQL, Scala, and PowerShell.
  • Experience developing batch and streaming ETL/ELT data pipelines.
  • Experience with Azure Data Factory, Azure Data Lake Storage, Azure SQL Database, and Azure Synapse.
  • Strong understanding of Apache Spark, Delta Lake, and distributed data processing.
  • Experience with relational SQL and NoSQL databases.
  • Knowledge of data modeling, schema design, and database architecture.
  • Experience with data governance, security, and Unity Catalog.
  • Experience optimizing Spark workloads, Databricks clusters, and cloud infrastructure.
  • Knowledge of Azure networking, VNETs, Key Vault, and identity management.
  • Experience with data orchestration, automation, and CI/CD processes.

Apply for this position

This job is hosted externally. Click below to view the full posting and apply.

Apply on www.dice.com
Prepare application

Good distractions

Loading talks and stories from around this role…