> Markdown version of [/jobs/ext/3971317-lead-data-engineer-with-databricks-r01570796](https://www.wearedevelopers.com/jobs/ext/3971317-lead-data-engineer-with-databricks-r01570796). Every page supports `.md` or `Accept: text/markdown`. Links point to the HTML versions so they work for humans too. Agent guide: [/agents.md](https://www.wearedevelopers.com/agents.md).

---

# Lead Data Engineer with Databricks - R01570796

- **Company:** Brillio
- **Location:** St. Louis, MO, United States
- **Experience:** Expert
- **Salary:** $140,000.0 - $150,000.0
- **Contract:** Permanent contract
- **Skills:** Unity 3d, Application Programming Interfaces (APIs), Amazon Web Services, Amazon S3, Automation of Tests, Microsoft Azure, Big Data, Cloud Computing, Cloud Storage, Code Review, Databases, Continuous Integration, Information Engineering, Extract Transform Load (ETL), Data Transformation, Distributed Computing Environment, Github, R (Programming Language), Identity and Access Management, Job Scheduling, Python (Programming Language), Performance Tuning, SQL Databases, Data Streaming, Scripting, Sql Optimization, Cloud Monitoring, Snowflake, Apache Spark, Caching, Data Lakes, Pyspark, Git Flow, Infrastructure Automation Frameworks, Terraform, Data Pipelines, Cisco, Key Vault, Jenkins, Databricks
- **Published:** October 27, 2026
- **Apply:** https://www.careerbuilder.com/job-details/lead-data-engineer-with-databricks-r01570796-saint-louis-mo--656f7724-ea95-4af5-8ddb-1436ad485948

## About the Role

R Programming Language, Fortune 1000 Customers, Customer Satisfaction, Database Extract Transform and Load (ETL), Python Programming/Scripting Language, Cisco Unity, Caching, Performance Tuning/Optimization, SQL (Structured Query Language), Amazon Simple Storage Service (S3), * Strong proficiency in Python and PySpark for distributed data processing and ETL.

 * Advanced SQL skills with experience in complex querying, window functions, and optimization.
 * Hands-on experience with Databricks (clusters, notebooks, Delta Lake, Unity Catalog, Delta Live Tables, workflows/jobs).
 * Proficiency in Terraform for infrastructure provisioning and management (Databricks resources, cloud storage, IAM, networking).
 * Experience with GitHub for version control and collaboration (branching, pull requests, code reviews).
 * Solid knowledge of CI/CD practices and tools, particularly Jenkins (pipelines, plugins for Databricks/GitHub/Terraform).
 * Working experience on Azure (Data Lake, Data Factory, Synapse, Key Vault, etc.) and/or AWS (S3, Glue, EMR, IAM, Lambda, etc.).
 * Understanding of big data concepts, data modeling (star/snowflake, dimensional), and lakehouse principles.
 * Familiarity with performance tuning in Spark/Databricks environments.

## Description

* Design, develop, and maintain ETL/ELT pipelines using PySpark and Python in Databricks (including notebooks, jobs, Delta Lake tables, and Unity Catalog for governance).
 * Implement medallion architecture (bronze/silver/gold layers) and optimize Spark jobs for performance, cost, scalability, and reliability (handling partitioning, skew, caching, adaptive query execution, etc.).
 * Write efficient SQL queries for data transformation, validation, and analytics within Databricks.
 * Provision and manage cloud infrastructure using Terraform (IaC) for Databricks workspaces, clusters, jobs, storage (ADLS/S3), networking, IAM roles/permissions, and related resources on Azure and/or AWS.
 * Implement and maintain CI/CD pipelines using Jenkins, GitHub (Actions/Repositories), and branching strategies for automated testing, deployment of notebooks, jobs, Delta Live Tables, and Terraform configurations.
 * Integrate data from diverse sources (databases, APIs, streaming, files) into cloud storage and processing layers.
 * Ensure data quality, lineage, security, and compliance (including Delta Lake ACID transactions, schema evolution, time travel, and access controls).
 * Monitor pipeline performance, troubleshoot failures, and implement alerting/observability (using Databricks tools, cloud monitoring services, or third-party solutions).
 * Optimize cloud costs through auto-scaling clusters, spot instances, job scheduling, and efficient resource usage.
 * Collaborate in agile teams, participate in code reviews, and contribute to best practices for data engineering.