> Markdown version of [/jobs/ext/469336-sr-data-engineer-sql-databricks-pyspark](https://www.wearedevelopers.com/jobs/ext/469336-sr-data-engineer-sql-databricks-pyspark). Every page supports `.md` or `Accept: text/markdown`. Links point to the HTML versions so they work for humans too. Agent guide: [/agents.md](https://www.wearedevelopers.com/agents.md). --- # Sr. Data Engineer (SQL / Databricks / PySpark) - **Company:** TalentOla View all jobs - **Location:** Chicago, IL, United States - **Experience:** Expert - **Salary:** $120,000.0 - $135,000.0 - **Contract:** Permanent contract - **Skills:** Java (Programming Language), .NET Framework, Application Programming Interfaces (APIs), Unit Testing, Microsoft Azure, Big Data, Cloud Database, Code Coverage, Code Review, Continuous Integration, Data Validation, Extract Transform Load (ETL), Software Debugging, Python (Programming Language), Microsoft SQL Server, SQL Azure, Performance Tuning, Azure Data Lake, SQL Databases, Systems Integration, TypeScript, Enterprise Data Management, Data Logging, Pulumi, Scripting, Azure Data Factory, Spring-boot, Data Lakes, AngularJS, Pyspark, Integration Tests, Data Pipelines, Azure Resource Manager, Databricks - **Published:** June 4, 2026 - **Apply:** https://www.careerjet.com/jobad/us74db85bac79047472edfe364ef4b970b ## About the Role * Engineering Degree BE/ME/BTech/MTech/BSc/MSc. * Technical certification in multiple technologies is desirable. Skills: - Mandatory skills * ETL / ELT Concepts: Strong understanding of pipeline patterns, incremental loads, data validation, and troubleshooting. * SQL: Advanced querying (CTEs, views, joins, complex query logic) and performance tuning for transformations and validation. Python: Production-quality development (modular code, testing, logging, integration with APIs/files, CICD, Unit Test/Integration test automation, Code Coverage). * PySpark: Distributed transformations and performance optimization (joins, partitions, debugging), CICD, Unit Test/Integration test automation, Code Coverage. * Azure Data Factory (ADF): Build/operate ADF pipelines, parameterization, triggers, monitoring, retry/error handling; integrate with Databricks/ADLS. * Databricks: Develop and operationalize notebooks/jobs/workflows; Delta Lake patterns; basic cluster/job configuration. * Azure Fundamentals + Pulumi: Hands-on with ADLS Gen2, Azure Portal, Storage Explorer, Resource Groups, Azure SQL, and familiarity integrating with Azure OpenAI. Able to use/maintain Pulumi scripts for provisioning and managing Azure resources across environments * Local to Chicago, IL required as this needs candidates to go for in person client interview and 3 days Work from Office per Week. Nice to have skills: - * Ability to support/translate validation rules with SQL scripts and create data quality reports. TypeScript: Useful for pulumi pipeline to create Azure components. * Java: Useful for integration with existing services/components. * .NET: Useful for integration with existing services/components. Angular / Spring Boot: Minor troubleshooting or coordination with app teams., Trading Operations Engineer, Need Scripting and SQL, Chicago, IL We are seeking a motivated team player with technical aptitude and market awareness to support the operation of a… + 2 months ago ## Description Seeking a Senior Data Engineer with deep expertise in SQL Server, Azure SQL, and Databricks to design, optimize, and deliver scalable cloud-based data solutions supporting analytics and enterprise data platforms. Roles and Responsibilities * Design, build, and support end-to-end data pipelines (ingestion, transformation, validation, publishing). Develop and optimize SQL and PySpark/Databricks transformations for large datasets. * Build production-grade Python components (reusable modules, logging, error handling, testing). Create and maintain Azure Data Factory (ADF) pipelines (triggers, parameterization, monitoring, failure handling). * Work within Azure environments (ADLS Gen2, Azure SQL, resource groups, portal operations). * Provision and maintain Azure components using Pulumi (Infrastructure as Code). Participate in code reviews, documentation, and operational support (triage + root cause analysis) ## Related Videos - [PySpark - Combining Machine Learning & Big Data](https://www.wearedevelopers.com/videos/44-pyspark-combining-machine-learning-big-data) - [Alibaba Big Data and Machine Learning Technology](https://www.wearedevelopers.com/videos/37-alibaba-big-data-and-machine-learning-technology) - [Why segmenting your infrastructure into tiers makes your infrastructure design better](https://www.wearedevelopers.com/videos/1960-why-segmenting-your-infrastructure-into-tiers-makes-your-infrastructure-design-better) - [Enjoying SQL data pipelines with dbt](https://www.wearedevelopers.com/videos/823-enjoying-sql-data-pipelines-with-dbt) - [Empowering Retail Through Applied Machine Learning](https://www.wearedevelopers.com/videos/976-empowering-retail-through-applied-machine-learning) - [Data Science in Retail](https://www.wearedevelopers.com/videos/586-data-science-in-retail) ## Related Articles - [Data Engineer Salary UK](https://www.wearedevelopers.com/magazine/253-data-engineer-salary-uk) - [Software Engineer Salary London](https://www.wearedevelopers.com/magazine/252-software-engineer-salary-london) - [Top-Paying Tech Jobs (with Salaries)](https://www.wearedevelopers.com/magazine/372-top-paying-tech-jobs-with-salaries) - [Where To Find Software Engineering Jobs](https://www.wearedevelopers.com/magazine/396-where-to-find-software-engineering-jobs) - [The Most Popular IT Jobs on the Market](https://www.wearedevelopers.com/magazine/376-the-most-popular-it-jobs-on-the-market) - [Making Data Warehouses Fast: A Developer’s Story](https://www.wearedevelopers.com/magazine/107-making-data-warehouses-fast-a-developer-s-story)