Azure Senior Data Lead leads modernization of Python

Aroha Technologies

New York, NY, United States

1 day ago

Apply on www.dice.com

Prepare application

Role details

Contract type

Permanent contract

Employment type

Full-time (> 32 hours)

Experience level

Expert

Experience required

7 years minimum

Working hours

Regular working hours

Job source

www.dice.com

Tech stack

Microsoft Azure Cloud Engineering Information Engineering Extract Transform Load (ETL) Data Systems Software Design Patterns Distributed Systems Apache Hive Python (Programming Language) Microsoft SQL Server Object-Oriented Software Development Oracle (Applications)

+16 more

Oracle SQL Developer Performance Tuning Standard Sql Azure Data Lake Software Engineering SQL Databases Azure Data Factory Apache Spark Caching Fastapi Data Lakes Pyspark Data Lakehouse Azure Synapse Analytics Data Pipelines Databricks

Job description

Azure Senior Data Lead leads modernization of Python applications into scalable PySpark solutions on Azure Databricks. Role Purpose

Lead the modernization and migration of existing Python object-oriented applications into scalable PySpark and Spark SQL data-processing solutions on Azure Databricks - bringing a strong blend of software engineering, data engineering, cloud architecture, and performance optimization., * Analyze existing Python OOP applications and redesign single-node processing logic for distributed Spark execution.

Design, develop, and deploy enterprise-scale data pipelines on Azure Databricks; build reusable PySpark frameworks and utility modules.
Implement Delta Lake solutions using the Bronze Silver Gold architecture.
Build robust ETL/ELT pipelines with Azure Data Factory, ADLS Gen2, and Azure Synapse Analytics.
Implement data quality, reconciliation, validation, and monitoring frameworks.
Optimize Spark jobs using partitioning, bucketing, caching, broadcast joins, Adaptive Query Execution, and Delta optimization.
Benchmark converted applications against original Python implementations.

Requirements

Azure Senior Data Lead with Azure Data Factory (ADF), Azure Databricks, SQL, Oracle PL/SQL, Python

Location: New York, NY

Experience: 7-12 Years

Primary Skills: Azure Data Factory (ADF), Azure Databricks, SQL, Oracle PL/SQL, Python

Certifications: Relevant certifications in Azure Data Factory, Azure Databricks, SQL, Oracle, or Python would be a plus., * Python (expert), OOP, and advanced Python design patterns

PySpark, Spark SQL, and SQL
Azure Databricks, Azure Data Factory, ADLS Gen2
Apache Spark, Delta Lake, Data Lakehouse architecture, distributed computing

Must-have Skills

Python
Azure Databricks
Azure Data Factory (ADF)
MS SQL
Oracle PL/SQL

Good to Have

PySpark
Certifications in Azure Data Factory, Azure Databricks, SQL, Oracle, or Python

Experience & Expected Outcome

Senior data engineering leader with proven delivery of large-scale Databricks modernization programs.

Expected outcome: Existing Python applications converted into scalable, cost-efficient, enterprise-grade data solutions on Azure Databricks with proven performance parity.