Data Engineer with Azure Databricks

Propertyvalue Quantum Technologies Llc

Warren, NJ, United States

12 days ago

Apply on www.dice.com

Prepare application

Role details

Contract type

Temporary contract

Employment type

Full-time (> 32 hours)

Experience level

Expert

Compensation

$176,800.0

Working hours

Regular working hours

Job source

www.dice.com

Tech stack

Unity 3d Data Analysis Microsoft Azure Continuous Integration Data Architecture Information Engineering Data Governance Extract Transform Load (ETL) Distributed Computing Environment Log Analysis Performance Tuning Azure Data Lake

+9 more

Scala (Programming Language) Azure Data Factory Sql Optimization Apache Spark Data Lakes Azure Synapse Analytics Data Pipelines Key Vault Databricks

Job description

Lead the architecture, design, and delivery of scalable enterprise data platforms using Azure Databricks, Delta Lake, and lakehouse patterns. Build and optimize reliable batch and real-time data pipelines for analytics, reporting, and Al workloads. Establish reusable engineering frameworks, coding standards, CI/CD practices, and performance and cost-optimization strategies. Implement data governance, security, lineage, and quality controls, including Unity Catalog capabilities. Provide technical leadership, mentor engineering teams, and collaborate with business, architecture, analytics, and data science stakeholders.

Requirements

Azure Databricks and Lakehouse: Databricks platform, Delta Lake, Unity Catalog, and enterprise lakehouse architecture. Data Engineering: Apache Spark, BySpark or Scala, advanced SQL, ETL/ELT design, and distributed processing Azure Data Services: Azure Data Factory, ADLS Gen2, Synapse Analytics, Key Vault, Monitor, and Log Analytics. Architecture and Governance: Data modeling, security, data quality, lineage, scalability, and performance optimization. Engineering Leadership: Azure DevOps, CI/CD, Agile delivery, stakeholder management, mentoring, and technical roadmap development.