Data Engineer - Azure Databricks & SQL Server

E-Solutions

Chicago, IL, United States

about 2 months ago

Role details

Contract type

Permanent contract

Employment type

Full-time (> 32 hours)

Experience level

Expert

Working hours

Regular working hours

Job source

dice.com

Tech stack

Application Programming Interfaces (APIs) Automation of Tests Microsoft Azure Code Coverage Continuous Integration Data Validation Extract Transform Load (ETL) Software Debugging Distributed Computing Environment Python (Programming Language) Microsoft SQL Server SQL Azure

+13 more

Performance Tuning Query Optimization Azure Data Lake Data Logging Pulumi Azure Data Factory Sql Optimization Data Lakes Pyspark Integration Tests Infrastructure Automation Frameworks Data Pipelines Databricks

Job description

Senior Data Engineer - Azure Databricks & SQL Server(Onsite at Chicago,IL)1SQL,Azure,Python,etl,Pyspark,Data Engineer,azure data factory,Databricks,adfN/AC2CUnited States

Requirements

Strong knowledge of ETL/ELT concepts: pipeline design, incremental loads, data validation, troubleshooting.
Advanced SQL: CTEs, joins, views, query optimization, performance tuning.
Python: Production-grade coding, APIs, testing, logging, CI/CD, unit/integration testing, code coverage.
PySpark: Distributed data processing, performance optimization, debugging, CI/CD, automated testing.
Azure Data Factory (ADF): Pipeline development, parameterization, triggers, monitoring, error handling, integration with Databricks & ADLS.
Databricks: Notebooks, jobs/workflows, Delta Lake, cluster/job configuration.
Azure: ADLS Gen2, Azure Portal, Storage Explorer, Azure SQL, Azure OpenAI integration, Pulumi for infrastructure provisioning.