> Markdown version of [/jobs/ext/609017-senior-databricks-engineer](https://www.wearedevelopers.com/jobs/ext/609017-senior-databricks-engineer). Every page supports `.md` or `Accept: text/markdown`. Links point to the HTML versions so they work for humans too. Agent guide: [/agents.md](https://www.wearedevelopers.com/agents.md). --- # Senior Databricks Engineer - **Company:** EXL SERVICE - **Location:** United States (Remote available) - **Experience:** Expert - **Contract:** Permanent contract - **Skills:** Airflow, Amazon Web Services, Microsoft Azure, Big Data, Cloud Computing, Cluster Analysis, Continuous Integration, Information Engineering, Data Governance, Data Integrity, Extract Transform Load (ETL), Data Security, Dataspaces, Data Vault Modeling, Software Design Patterns, DevOps, Dimensional Modeling, Monitoring of Systems, Python (Programming Language), Raw Data, DataOps, SQL Databases, Data Streaming, Workflow Management Systems, Azure Service Bus, Grafana, Apache Spark, Git, Data Lakes, Pyspark, Real Time Data, Apache Kafka, Machine Learning Operations, Terraform, Splunk, Data Pipelines, Serverless Computing, Databricks - **Published:** June 20, 2026 - **Apply:** https://www.indeed.com/viewjob?jk=8dce8d9e924a15a0 ## About the Role Do you have experience in Scalability?, * Qualifications: 6+ Years of hands-on, progressive experience in Data Engineering, with at least 5 years focused heavily on the Databricks platform. * Architectural Understanding: Expert knowledge of Medallion Architecture, Data Vault 2.0 or Dimensional Modeling, and modern Lakehouse design patterns. * Scale Expertise: Proven track record of building and managing large-scale data infrastructure (Petabyte-scale) in cloud-native environments. * Industry Experience: Experience in the Insurance or Financial Services industry is preferred (focusing on claims, policy, or risk data). * Technical Toolset: + Cloud Environment: Azure (preferred), AWS,. + Databricks Stack: Unity Catalog, Delta Live Tables, Databricks SQL, MLflow. + Core Languages: Expert-level SQL, Python, and PySpark. + Supporting Tools: dbt (Databricks adapter), Git, and Orchestration tools ## Description Job Description: We are seeking a high-caliber Senior Databricks Engineer to lead the architecture, development, and optimization of our next-generation Lakehouse platform. This is a critical role for a technical leader with 6+ years of deep data engineering expertise, specifically within the Databricks ecosystem. The ideal candidate will drive technical direction, establish robust data governance, and deliver high-impact, scalable data solutions that bridge the gap between raw data and actionable business intelligence., Responsibilities: Data Pipeline Development & Management * Ingestion & Transformation: Design and optimize high-volume ETL/ELT pipelines using Delta Live Tables (DLT) and PySpark, ensuring data integrity across the Bronze, Silver, and Gold layers. * Workflow Orchestration: Develop and maintain sophisticated pipelines using Databricks Workflows or Airflow, focusing on modularity, reusability, and automated error handling. * Streaming & Real-time Integration: Implement real-time data flows utilizing Structured Streaming and Kafka/Event Hubs to enable immediate data availability for downstream consumption. * Data Security & Privacy: Enforce data anonymization and fine-grained access controls to ensure compliance with global regulations (GDPR/CCPA/HIPAA). * DataOps & DevOps: Implement CI/CD patterns using Databricks Asset Bundles (DABs), Terraform, and Git to automate environment parity and deployments. Data Ecosystem Management & Monitoring * Open Table Formats: Manage and optimize Delta Lake storage, utilizing advanced features like Liquid Clustering, Z-Ordering, and Change Data Feed (CDF). * Compute Engine Optimization: Drive cost efficiency and performance by optimizing Spark configurations, Photon engine utilization, and Serverless SQL Warehouses. * Observability & Monitoring: Integrate comprehensive monitoring and alerting (e.g., Databricks System Tables, Grafana, or Splunk) to rapidly identify bottlenecks and troubleshoot production issues. ## Related Videos - [From DevOps to Scaled DevOps: How We’re Rebuilding Continuous Delivery as a Platform](https://www.wearedevelopers.com/videos/100018-from-devops-to-scaled-devops-how-we-re-rebuilding-continuous-delivery-as-a-platform) - [Our journey with Spring Boot in a microservice architecture](https://www.wearedevelopers.com/videos/511-our-journey-with-spring-boot-in-a-microservice-architecture) - [How a Small Team Shrank a Microsoft Monorepo by 94%](https://www.wearedevelopers.com/videos/1236-how-a-small-team-shrank-a-microsoft-monorepo-by-94) - [From Messy Queries to Scalable Systems - How Data Engineering actually works](https://www.wearedevelopers.com/videos/100203-from-messy-queries-to-scalable-systems-how-data-engineering-actually-works) - [DevOps Maturity Check – a way to balance autonomy and alignment](https://www.wearedevelopers.com/videos/58-devops-maturity-check-a-way-to-balance-autonomy-and-alignment) - [Modern Data Architectures need Software Engineering](https://www.wearedevelopers.com/videos/1030-modern-data-architectures-need-software-engineering) ## Related Articles - [Data Engineer Salary UK](https://www.wearedevelopers.com/magazine/253-data-engineer-salary-uk) - [Top Big Data Technologies That You Need to Know](https://www.wearedevelopers.com/magazine/108-top-big-data-technologies-that-you-need-to-know) - [Highest Paying Tech Companies for Developers](https://www.wearedevelopers.com/magazine/220-highest-paying-tech-companies-for-developers) - [Making Data Warehouses Fast: A Developer’s Story](https://www.wearedevelopers.com/magazine/107-making-data-warehouses-fast-a-developer-s-story) - [Data Analyst Salary in the UK](https://www.wearedevelopers.com/magazine/278-data-analyst-salary-in-the-uk) - [Fully Remote Software Engineer Jobs](https://www.wearedevelopers.com/magazine/447-fully-remote-software-engineer-jobs)