> Markdown version of [/jobs/ext/1796885-databricks-data-engineer](https://www.wearedevelopers.com/jobs/ext/1796885-databricks-data-engineer). Every page supports `.md` or `Accept: text/markdown`. Links point to the HTML versions so they work for humans too. Agent guide: [/agents.md](https://www.wearedevelopers.com/agents.md). --- # Databricks Data Engineer - **Company:** EXL SERVICE - **Location:** United States (Remote available) - **Experience:** Expert - **Salary:** $65,000.0 - $87,000.0 - **Contract:** Permanent contract - **Skills:** Unity 3d, Agile Methodology, Artificial Intelligence, Data Analysis, Microsoft Azure, Code Review, Continuous Integration, Information Engineering, Data Governance, Data Integration, Extract Transform Load (ETL), Data Security, Data Warehousing, Apache Hive, JSON, Python (Programming Language), Key Management, Meta-Data Management, Performance Tuning, Systems Development Life Cycle, Role-Based Access Control, Azure Data Lake, SQL Databases, Data Streaming, Extensible Markup Language (XML), Azure Data Factory, Delivery Pipeline, Snowflake, Apache Spark, Caching, Ab Initio, Data Lakes, Pyspark, Information Technology, Deployment Automation, Restful APIs, Azure Synapse Analytics, Data Pipelines, Databricks, Web Api - **Published:** July 29, 2026 - **Apply:** https://fa-ewjt-saasfaprod1.fa.ocs.oraclecloud.com/hcmUI/CandidateExperience/en/sites/CX_2/jobs/preview/17678 ## About the Role Technical Expertise * 10-12 years of experience in Data Engineering, ETL Development, and Data Warehousing. * Hands-on Databricks development experience. * Strong experience with Spark, Delta Lake, Unity Catalog, Databricks Workflows, and SQL Warehouses. * Strong experience with PySpark, Spark SQL, SQL, and Python. * Experience building enterprise ETL/ELT pipelines using Databricks and Azure Data Platform. * Experience with Azure Data Lake Storage (ADLS Gen2), Azure Synapse, Azure Key Vault, and Azure DevOps. * Experience implementing CDC, SCD, incremental loading, and data quality frameworks. * Experience integrating REST APIs and processing JSON/XML datasets. * Experience with Git, CI/CD, release management, and deployment automation. * Strong knowledge of performance tuning, partitioning, caching, broadcast joins, and Spark optimization. * Experience working with healthcare data platforms is preferred. * Microsoft Azure certifications (e.g., DP-203 Azure Data Engineer Associate) or Databricks certifications (Data Engineer Associate/Professional). * Experience with Ab Initio suite of products is also preferred. ## Description EXL Service is seeking an accomplished Senior Databricks Data Engineer with 10-12 years of experience designing, developing, and optimizing enterprise data platforms and large-scale ETL solutions. The ideal candidate brings deep expertise in Databricks, Spark, Delta Lake, Azure Data Platform, and modern data engineering practices. This role is responsible for building scalable data pipelines, implementing cloud-native data solutions, improving platform performance, and delivering reliable data products for analytics, reporting, and AI/ML initiatives. Base Compensation Range: 65,000 - 87,000, Data Engineering & Platform Development * Design, develop, and optimize enterprise-scale data pipelines using Databricks, Spark, Delta Lake, and Azure Data Lake Storage. * Build batch and incremental ETL/ELT pipelines, real-time streaming architectures, and reusable data integration frameworks. * Develop scalable ingestion frameworks supporting structured, semi-structured, and API-based data sources. * Build reusable data engineering frameworks for ingestion, transformation, validation, reconciliation, and publishing. * Develop Delta Lake solutions using partitioning, optimization, Z-Ordering, Liquid Clustering, and performance tuning techniques. * Implement CDC, incremental processing, merge strategies, and data synchronization across enterprise platforms. * Develop Databricks Workflows, notebooks, SQL Jobs, and automation for production workloads. * Integrate external REST APIs and process JSON/XML data into analytics-ready datasets. * Implement robust data quality checks, audit frameworks, monitoring, and error handling. * Optimize Spark jobs for performance, scalability, and cost efficiency. Databricks & Azure Development * Develop solutions using Azure Data Lake Storage Gen2, Databricks, Unity Catalog, Azure Key Vault, Azure DevOps, and Azure Synapse. * Build secure data pipelines using Unity Catalog, RBAC, service principals, and managed identities. * Develop reusable notebook frameworks using PySpark and Spark SQL. * Implement CI/CD deployment pipelines using Azure DevOps. * Manage environment promotion across Development, QA, UAT, and Production. * Troubleshoot production issues and optimize workloads for reliability and scalability. Data Integration & Analytics * Design enterprise data models supporting reporting, analytics, and downstream applications. * Develop healthcare and financial data integration pipelines supporting multiple source systems. * Build reusable metadata-driven ETL frameworks. * Integrate third-party APIs including NLP, terminology normalization, and identity resolution services. * Support data governance, lineage, and metadata management initiatives., Design, build, and migrate large-scale batch data pipelines into Databricks. Develop and optimize production-grade Python/Spark/SQL solutions, support parallel legacy and cloud systems during migration, lead technical design discussions, use AI tools to improve workflows, ensure pipeline reliability through testing/monitoring/tuning, and participate in a rotating on-call schedule. Top Skills: Anthropic ClaudeSparkAzure Data FactoryAzure DevopsCi/CdCosmos DbDatabricksDelta LakeEvent HubsGitGithub CopilotJenkinsNoSQLPythonSQLSynapse AnalyticsTeradata Velera ## Related Videos - [PySpark - Combining Machine Learning & Big Data](https://www.wearedevelopers.com/videos/44-pyspark-combining-machine-learning-big-data) - [Web APIs you might not know about](https://www.wearedevelopers.com/videos/281-web-apis-you-might-not-know-about) - [Tips and Tricks for Working with JSON](https://www.wearedevelopers.com/videos/1229-tips-and-tricks-for-working-with-json) - [From Messy Queries to Scalable Systems - How Data Engineering actually works](https://www.wearedevelopers.com/videos/100203-from-messy-queries-to-scalable-systems-how-data-engineering-actually-works) - [Data Science in Retail](https://www.wearedevelopers.com/videos/586-data-science-in-retail) - [Empowering Retail Through Applied Machine Learning](https://www.wearedevelopers.com/videos/976-empowering-retail-through-applied-machine-learning) ## Related Articles - [Data Engineer Salary UK](https://www.wearedevelopers.com/magazine/253-data-engineer-salary-uk) - [Making Data Warehouses Fast: A Developer’s Story](https://www.wearedevelopers.com/magazine/107-making-data-warehouses-fast-a-developer-s-story) - [Top Big Data Technologies That You Need to Know](https://www.wearedevelopers.com/magazine/108-top-big-data-technologies-that-you-need-to-know) - [Highest Paying Tech Companies for Developers](https://www.wearedevelopers.com/magazine/220-highest-paying-tech-companies-for-developers) - [Data Analyst Salary in the UK](https://www.wearedevelopers.com/magazine/278-data-analyst-salary-in-the-uk) - [What Are The Top Skills Required For Azure Developers?](https://www.wearedevelopers.com/magazine/77-what-are-the-top-skills-required-for-azure-developers)