Data Engineer - Azure Databricks & SQL Server

E-Solutions
Chicago, United States of America
7 days ago

Role details

Contract type
Permanent contract
Employment type
Full-time (> 32 hours)
Working hours
Regular working hours
Languages
English
Experience level
Senior

Job location

Chicago, United States of America

Tech stack

API
Automation of Tests
Azure
Code Coverage
Continuous Integration
Data Validation
ETL
Software Debugging
Distributed Computing Environment
Python
Microsoft SQL Server
SQL Azure
Performance Tuning
Query Optimization
Azure
Data Logging
Pulumi
Azure
Sql Optimization
Data Lake
PySpark
Integration Tests
Infrastructure Automation Frameworks
Data Pipelines
Databricks

Job description

Senior Data Engineer - Azure Databricks & SQL Server(Onsite at Chicago,IL)1SQL,Azure,Python,etl,Pyspark,Data Engineer,azure data factory,Databricks,adfN/AC2CUnited States

Requirements

  • Strong knowledge of ETL/ELT concepts: pipeline design, incremental loads, data validation, troubleshooting.
  • Advanced SQL: CTEs, joins, views, query optimization, performance tuning.
  • Python: Production-grade coding, APIs, testing, logging, CI/CD, unit/integration testing, code coverage.
  • PySpark: Distributed data processing, performance optimization, debugging, CI/CD, automated testing.
  • Azure Data Factory (ADF): Pipeline development, parameterization, triggers, monitoring, error handling, integration with Databricks & ADLS.
  • Databricks: Notebooks, jobs/workflows, Delta Lake, cluster/job configuration.
  • Azure: ADLS Gen2, Azure Portal, Storage Explorer, Azure SQL, Azure OpenAI integration, Pulumi for infrastructure provisioning.

Apply for this position