Azure Data Engineer (ADF & Databricks)

Microsoft Windows Azure, Microsoft Windows Azure, Error Handling, Data Management, Data Mapping, SQL (Structured Query Language), Data Sets, Agile Programming Methodologies, Git, Pytest, Azure Data Factory Pipelines, data flows (mapping wrangling), triggers, integration runtimes, parameterization, error handling & retries.

Azure Databricks Databricks Notebooks, Jobs Workflows, PySpark Scala, Delta Lake, cluster configuration and Eligible to workimization.

SQL Expert level SQL (CTEs, window functions, query tuning), working with large datasets.

Nice to have skills

Agile

Infrastructure as Code exposure is a plus.

Detailed Job Description

Experience 6-8 years in data engineering with at least 3 years on Azure. Azure Data Factory Pipelines, data flows mapping wrangling, triggers, integration runtimes, parameterization, error handling retries. Azure Databricks Databricks Notebooks, Jobs Workflows, PySpark Scala, Delta Lake, cluster configuration and Eligible to workimization. Programming Advanced Python or Scala at least one, strong coding standards, modularization, and unit testing e.g., pytest. SQL Expert level SQL CTEs, window functions, query tuning, working with large datasets. Data Storage Formats ADLS Gen2, Delta Lake, Parquet, partitioning, ZOrdering, Eligible to workIMIZE VACUUM. Ops CICD Git, branching strategies, CICD for ADF Databricks environment promotion Infrastructure as Code exposure is a plus.

Minimum years of experience: 8-10 years

Certifications Needed :No

Top 3 responsibilities you would expect the Subcon to shoulder and execute

Coding and Development

UI testing

Stakeholder management

Security Notice: Our official website is www.diverselynx.com We do not operate or authorize any other websites representing Diverse Lynx LLC.