> Markdown version of [/jobs/ext/3961980-senior-databricks-engineer](https://www.wearedevelopers.com/jobs/ext/3961980-senior-databricks-engineer). Every page supports `.md` or `Accept: text/markdown`. Links point to the HTML versions so they work for humans too. Agent guide: [/agents.md](https://www.wearedevelopers.com/agents.md).

---

# Senior Databricks Engineer

- **Company:** Q1 Technologies, Inc.
- **Location:** Aurora, IL, United States
- **Experience:** Expert
- **Contract:** Permanent contract
- **Skills:** Unity 3d, Application Programming Interfaces (APIs), Artificial Intelligence, Amazon Web Services, Amazon S3, Automation of Tests, Software Code Optimization, Databases, System Configuration, Continuous Integration, Data Governance, Extract Transform Load (ETL), Data Security, Data Warehousing, Distributed Computing Environment, Apache Hive, Identity and Access Management, Python (Programming Language), Key Management, Machine Learning, Performance Tuning, Standard Sql, Search Technologies, SQL Databases, Systems Integration, S3 Bucket, Data Processing, Large Language Models, Apache Spark, Git, Data Lakes, Pyspark, Git Flow, Data Lineage, AWS Glue, AWS Data Analytics, Machine Learning Operations, Software Version Control, Data Pipelines, Cisco, Amazon Redshift, Databricks
- **Published:** October 27, 2026
- **Apply:** https://www.careerbuilder.com/job-details/senior-databricks-engineer-aurora-il--01386d54-2d45-4c0d-9531-c3f097b8073b

## About the Role

Apache Spark, Application Programming Interface (API), Performance Tuning/Optimization, Data Processing, SQL (Structured Query Language), Database Extract Transform and Load (ETL), Cisco Unity, Amazon Simple Storage Service (S3), Data Warehousing, Amazon Web Services (AWS)

## Description

Apache Spark: Mastery of Spark SQL and the DataFrame API for distributed data processing, performance tuning, and optimizing execution plans.
Delta Lake & Lakehouse: Building medallion architectures (Bronze, Silver, Gold layers) and using optimizations like Z-Ordering and ACID transactions
Pipelines & Orchestration: Developing with Delta Live Tables (DLT) and Databricks Workflows to design resilient, idempotent ELT/ETL pipelines.
Unity Catalog: Configuring data governance, row/column-level security, and data lineage
Languages: Advanced proficiency in Python (PySpark) and SQL
Storage & Data Lakes: Expert in reading/writing to Amazon S3 and integrating with AWS data warehouses (e.g., Amazon Redshift).
Security & IAM: Implementing cross-account IAM roles, S3 bucket policies, and Customer-Managed Keys (CMK) via AWS KMS
.Compute Management: Managing cluster policies, instance profiles, and utilizing Spot Instances to optimize Databricks compute costs on AWS.
Ecosystem Integration: Familiarity with integrating Databricks jobs alongside native services like AWS Glue, Amazon Kinesis, and AWS Step Functions
CI/CD & Version Control: Git-based workflows using Databricks Git Folders and managing deployments.
AI & MLOps: Using MLflow for experiment tracking and model registries.
Familiarity with LLMs, Vector Search, and GenAI integrations.

Roles & Responsibilities

Data Pipeline Development: Extract, transform, and load (ETL) data from multiple sources, building batch and streaming pipelines.Spark & Code Optimization: Write highly efficient PySpark, Scala, or SQL code. Troubleshoot distributed processing bottlenecks and optimize job performance.
Lakehouse Management: Work with the Medallion architecture to transition data through Bronze (raw), Silver (cleansed), and Gold (aggregated) layers in Delta Lake.
Orchestration & Automation: Schedule and automate workflows using Databricks Jobs, Workflows, or Delta Live Tables (DLT). Configure automated tests, retries, and error handling.
Governance & Security: Implement data privacy and access controls utilizing the Databricks Unity Catalog, ensuring compliance and secure data sharing.
Collaboration: Work alongside Data Scientists, Data Analysts, and ML Engineers to prepare data features and support BI reporting, ML training, and generative AI initiatives.