Sr. Data Engineer (SQL / Databricks / PySpark)

Seeking a Senior Data Engineer with deep expertise in SQL Server, Azure SQL, and Databricks to design, optimize, and deliver scalable cloud-based data solutions supporting analytics and enterprise data platforms. Roles and Responsibilities

• Design, build, and support end-to-end data pipelines (ingestion, transformation, validation, publishing). Develop and optimize SQL and PySpark/Databricks transformations for large datasets.
• Build production-grade Python components (reusable modules, logging, error handling, testing). Create and maintain Azure Data Factory (ADF) pipelines (triggers, parameterization, monitoring, failure handling).
• Work within Azure environments (ADLS Gen2, Azure SQL, resource groups, portal operations).
• Provision and maintain Azure components using Pulumi (Infrastructure as Code). Participate in code reviews, documentation, and operational support (triage + root cause analysis)

• Engineering Degree BE/ME/BTech/MTech/BSc/MSc.
• Technical certification in multiple technologies is desirable.

Skills: - Mandatory skills

• ETL / ELT Concepts: Strong understanding of pipeline patterns, incremental loads, data validation, and troubleshooting.
• SQL: Advanced querying (CTEs, views, joins, complex query logic) and performance tuning for transformations and validation. Python: Production-quality development (modular code, testing, logging, integration with APIs/files, CICD, Unit Test/Integration test automation, Code Coverage).
• PySpark: Distributed transformations and performance optimization (joins, partitions, debugging), CICD, Unit Test/Integration test automation, Code Coverage.
• Azure Data Factory (ADF): Build/operate ADF pipelines, parameterization, triggers, monitoring, retry/error handling; integrate with Databricks/ADLS.
• Databricks: Develop and operationalize notebooks/jobs/workflows; Delta Lake patterns; basic cluster/job configuration.
• Azure Fundamentals + Pulumi: Hands-on with ADLS Gen2, Azure Portal, Storage Explorer, Resource Groups, Azure SQL, and familiarity integrating with Azure OpenAI. Able to use/maintain Pulumi scripts for provisioning and managing Azure resources across environments
• Local to Chicago, IL required as this needs candidates to go for in person client interview and 3 days Work from Office per Week.

Nice to have skills: -

• Ability to support/translate validation rules with SQL scripts and create data quality reports. TypeScript: Useful for pulumi pipeline to create Azure components.
• Java: Useful for integration with existing services/components.
• .NET: Useful for integration with existing services/components. Angular / Spring Boot: Minor troubleshooting or coordination with app teams., Trading Operations Engineer, Need Scripting and SQL, Chicago, IL We are seeking a motivated team player with technical aptitude and market awareness to support the operation of a…

• 2 months ago

• $120,000-135,000 per year We're building a relationship-oriented bank for the modern world. We need talented, passionate professionals who are dedicated to doing what's right for our clients. At CIBC, we …

• 1 month ago

*, + $150,000-170,000 per year We're building a relationship-oriented bank for the modern world. We need talented, passionate professionals who are dedicated to doing what's right for our clients. At CIBC, we …

• 2 months ago