> Markdown version of [/jobs/ext/2835795-lead-data-engineer](https://www.wearedevelopers.com/jobs/ext/2835795-lead-data-engineer). Every page supports `.md` or `Accept: text/markdown`. Links point to the HTML versions so they work for humans too. Agent guide: [/agents.md](https://www.wearedevelopers.com/agents.md). --- # Lead Data Engineer - **Company:** UST Inc - **Location:** Chicago, IL, United States (Remote available) - **Experience:** Expert - **Salary:** $92,000.0 - $138,000.0 - **Contract:** Temporary contract - **Skills:** Airflow, Amazon Web Services, Amazon S3, Apache HTTP Server, Code Review, Data Architecture, Information Engineering, Data Governance, Data Infrastructure, Data Masking, Distributed Systems, Identity and Access Management, Interoperability, Python (Programming Language), Role-Based Access Control, Cloud Services, Workflow Management Systems, Enterprise Data Management, Sql Optimization, Fast Healthcare Interoperability Resources, Pyspark, Data Lineage, Data Analytics, Health Level Seven International, Data Pipelines, Databricks - **Published:** September 10, 2026 - **Apply:** https://jobs.localjobnetwork.com/apply/add/88287931/1 ## About the Role * 7+ years of data engineering experience * At least 2 years in a technical lead role * Production experience building and optimizing large-scale pipelines within Databricks * Production experience deploying data infrastructure on AWS (S3, EMR, Lambda, IAM, Glue) * Explicit, hands-on experience processing Medicare Advantage data (CMS enrollment, RAPS/EDPS submissions, MMR/MOR files, HCC risk adjustment models) * Proven experience working with healthcare interoperability standards (FHIR and HL7 models) * Proven experience implementing enterprise-level Role-Based Access Control (RBAC) and row-level security for PHI * Skills & Technical Competencies: * Mastery of Python, PySpark, and advanced SQL for complex distributed computing tasks * Deep understanding and practical experience with Apache Iceberg (open table formats) * Expert-level knowledge of Databricks Unity Catalog for centralized data governance * Comprehensive understanding of US Healthcare data architecture * Strong data pipeline design and optimization capabilities * Security and compliance implementation expertise * Knowledge & Domain Expertise: * Deep domain expertise in US Healthcare systems * In-depth knowledge of Medicare Advantage programs and workflows * Understanding of healthcare regulations and HIPAA compliance requirements * Knowledge of HL7 and FHIR healthcare interoperability standards * Preferred Qualifications & Certifications: * Databricks Certified Data Engineer Professional * AWS Certified Data Analytics certification * Familiarity with orchestration tools like Apache Airflow or AWS Step Functions ## Description UST is searching for a Lead Data Engineer to architect, build, and optimize scalable, cloud-native data systems. In this role, you will lead the development of enterprise data platforms using Databricks and AWS. The opportunity: * Lead the design, implementation, and maintenance of robust data pipelines on AWS using Databricks * Architect scalable lakehouse structures leveraging Apache Iceberg to ingest, process, and store complex healthcare data types * Establish comprehensive data governance and data lineage frameworks using Databricks Unity Catalog * Implement secure access controls, data masking, and row-level security through Role-Based Access Control (RBAC) * Architect and scale ingestion frameworks capable of processing HL7 and FHIR data feeds natively into the lakehouse ecosystem * Oversee end-to-end processing of Medicare Advantage data assets, ensuring accurate risk adjustment and reporting workflows * Ensure strict adherence to healthcare regulations, including HIPAA compliance and robust data governance frameworks * Mentor junior engineers, conduct code reviews, and drive technical best practices across the data team This position description identifies the responsibilities and tasks typically associated with the performance of the position. Other relevant essential functions may be required. ## Related Videos - [From Messy Queries to Scalable Systems - How Data Engineering actually works](https://www.wearedevelopers.com/videos/100203-from-messy-queries-to-scalable-systems-how-data-engineering-actually-works) - [PySpark - Combining Machine Learning & Big Data](https://www.wearedevelopers.com/videos/44-pyspark-combining-machine-learning-big-data) - [Data Science in Retail](https://www.wearedevelopers.com/videos/586-data-science-in-retail) - [WeAreDevelopers LIVE - CSS is DOOMed](https://www.wearedevelopers.com/videos/1838-wearedevelopers-live-css-is-doomed) - [Empowering Retail Through Applied Machine Learning](https://www.wearedevelopers.com/videos/976-empowering-retail-through-applied-machine-learning) - [Enjoying SQL data pipelines with dbt](https://www.wearedevelopers.com/videos/823-enjoying-sql-data-pipelines-with-dbt) ## Related Articles - [Data Engineer Salary UK](https://www.wearedevelopers.com/magazine/253-data-engineer-salary-uk) - [Making Data Warehouses Fast: A Developer’s Story](https://www.wearedevelopers.com/magazine/107-making-data-warehouses-fast-a-developer-s-story) - [Top Big Data Technologies That You Need to Know](https://www.wearedevelopers.com/magazine/108-top-big-data-technologies-that-you-need-to-know) - [Data Analyst Salary in the UK](https://www.wearedevelopers.com/magazine/278-data-analyst-salary-in-the-uk) - [Highest Paying Tech Companies for Developers](https://www.wearedevelopers.com/magazine/220-highest-paying-tech-companies-for-developers) - [How We Built a Worry-Free System That Runs for 10+ Years – And What We’d Do Again](https://www.wearedevelopers.com/magazine/751-how-we-built-a-worry-free-system-that-runs-for-10-years-and-what-we-d-do-again)