> Markdown version of [/jobs/ext/3029579-senior-databricks-engineer](https://www.wearedevelopers.com/jobs/ext/3029579-senior-databricks-engineer). Every page supports `.md` or `Accept: text/markdown`. Links point to the HTML versions so they work for humans too. Agent guide: [/agents.md](https://www.wearedevelopers.com/agents.md). --- # Senior Databricks Engineer - **Company:** UST Inc - **Location:** Alpharetta, GA, United States - **Experience:** Expert - **Salary:** $106,000.0 - $159,000.0 - **Contract:** Temporary contract - **Skills:** Artificial Intelligence, Amazon Web Services, Amazon S3, Data Analysis, Apache HTTP Server, Continuous Integration, Data Architecture, Information Engineering, Data Governance, Data Security, Distributed Computing Environment, Fault Tolerance, Apache Hive, Identity and Access Management, Interoperability, Key Management, Machine Learning, Performance Tuning, Role-Based Access Control, Azure Machine Learning, Data Streaming, Systems Integration, Enterprise Data Management, Data Logging, Data Ingestion, Sql Optimization, Apache Spark, Data Lakes, Pyspark, Infrastructure Automation Frameworks, Data Lineage, Deployment Automation, Databricks - **Published:** September 22, 2026 - **Apply:** https://diversityjobs.com/candidate/trackback_apply?vacancyId=[vacid] ## About the Role * 5+ years of professional data engineering experience, with substantial recent hands-on experience developing production solutions on Databricks * Strong hands-on expertise with Apache Spark / PySpark and advanced SQL * Strong experience designing and implementing enterprise-scale batch and streaming data pipelines * Deep understanding of distributed data processing, partitioning, schema evolution, fault tolerance, performance optimization, and scalability * Strong experience with Delta Lake and lakehouse architecture * Hands-on experience with Apache Iceberg table format, including table design, partitioning, schema evolution, interoperability, and production implementation * Experience working with multiple enterprise source systems and downstream analytical, reporting, and ML platforms * Strong hands-on experience with Databricks Unity Catalog in an enterprise environment * Strong understanding of Databricks privilege hierarchies and permission inheritance * Experience implementing least-privilege access, segregation of duties, and environment-specific access controls across development, test/UAT, and production * Experience designing secure access patterns for both human users and machine/service identities * Ability to troubleshoot complex authorization and access-control issues in Databricks * Hands-on experience implementing data tokenization, masking, anonymization, or de-identification techniques for sensitive enterprise data * Strong understanding of protecting PHI, PII, and other sensitive healthcare information * Experience implementing encryption for data exchanged between platforms and services * Understanding of enterprise key-management concepts and secure key-handling practices * Experience designing pipelines that maintain appropriate security controls throughout ingestion, processing, storage, and consumption * Ability to balance data usability with privacy, security, and regulatory requirements * AWS * Strong experience working in AWS-based enterprise data environments * Hands-on experience integrating Databricks with Amazon S3 and other AWS services * Understanding of AWS IAM, security controls, networking, monitoring, logging, and enterprise cloud architecture * Experience working with AWS DevOps teams and automated deployment processes * Understanding of secure connectivity between Databricks and enterprise data sources/consumers * US Healthcare * Minimum 3 years of recent professional experience working in US healthcare * Hands-on experience working with healthcare data in an enterprise environment * Experience with clinical and/or claims data, with preference for candidates who have worked extensively with both * Understanding of healthcare data terminology, business processes, data quality challenges, and healthcare-specific business rules * Experience working with data containing PHI/PII * Practical understanding of HIPAA and healthcare data security/privacy requirements * Experience collaborating with Clinical SMEs, Claims SMEs, healthcare business stakeholders, and technology teams * Ability to translate healthcare business requirements into scalable and secure data engineering solutions ## Description UST is searching for a Senior Databricks Engineer with deep hands-on experience designing, developing, securing, and deploying complex enterprise data pipelines on AWS Databricks within the US healthcare industry. The engineer will build scalable batch and streaming data pipelines supporting clinical, claims, analytics, and machine learning use cases. The opportunity: * Design, develop, test, deploy, and support complex batch and streaming data pipelines using AWS Databricks * Design scalable lakehouse solutions using Delta Lake and Apache Iceberg table formats * Implement enterprise data governance using Databricks Unity Catalog, including catalogs, schemas, tables, views, volumes, external locations, and governed data access * Design and implement complex Databricks RBAC and access-control models across users, groups, service principals, workspaces, catalogs, schemas, tables, views, external locations, and other governed resources * Implement secure data-access patterns following least privilege, separation of duties, and role-based access control principles * Design and implement appropriate data tokenization, masking, encryption, and de-identification techniques for sensitive healthcare data, including PHI/PII * Ensure encryption is appropriately implemented for data at rest and data in transit, including integration with enterprise key-management and security controls where applicable * Develop high-volume data ingestion and transformation pipelines using PySpark, Spark SQL and Databricks-native capabilities * Develop and optimize both batch and real-time/streaming pipelines * Work with SMEs to understand healthcare data structures, business rules, data lineage, quality requirements, and transformation logic * Partner with Machine Learning Engineers to create secure, governed, high-quality datasets for ML/AI workloads * Collaborate with AWS DevOps engineers on CI/CD, infrastructure automation, deployment, monitoring, logging, and production operations This position description identifies the responsibilities and tasks typically associated with the performance of the position. Other relevant essential functions may be required. ## Related Videos - [Data Science in Retail](https://www.wearedevelopers.com/videos/586-data-science-in-retail) - [PySpark - Combining Machine Learning & Big Data](https://www.wearedevelopers.com/videos/44-pyspark-combining-machine-learning-big-data) - [WeAreDevelopers LIVE - CSS is DOOMed](https://www.wearedevelopers.com/videos/1838-wearedevelopers-live-css-is-doomed) - [Cutting LLM Costs Without Cutting Quality: How to Beat Proprietary LLMs with Fine-Tuned Open Source](https://www.wearedevelopers.com/videos/100151-cutting-llm-costs-without-cutting-quality-how-to-beat-proprietary-llms-with-fine-tuned-open-source) - [OLTP in the Lakehouse: Redefining Data for AI Workloads](https://www.wearedevelopers.com/videos/2038-oltp-in-the-lakehouse-redefining-data-for-ai-workloads) - [Empowering Retail Through Applied Machine Learning](https://www.wearedevelopers.com/videos/976-empowering-retail-through-applied-machine-learning) ## Related Articles - [Top Big Data Technologies That You Need to Know](https://www.wearedevelopers.com/magazine/108-top-big-data-technologies-that-you-need-to-know) - [Data Engineer Salary UK](https://www.wearedevelopers.com/magazine/253-data-engineer-salary-uk) - [Making Data Warehouses Fast: A Developer’s Story](https://www.wearedevelopers.com/magazine/107-making-data-warehouses-fast-a-developer-s-story) - [Data Analyst Salary in the UK](https://www.wearedevelopers.com/magazine/278-data-analyst-salary-in-the-uk) - [Coffee with Developers - Maria Apazoglou - Making AI understandable for all in production](https://www.wearedevelopers.com/magazine/475-coffee-with-developers-maria-apazoglou-making-ai-understandable-for-all-in-production) - [Highest Paying Tech Companies for Developers](https://www.wearedevelopers.com/magazine/220-highest-paying-tech-companies-for-developers)