> Markdown version of [/jobs/ext/2219504-aws-databricks-data-engineer](https://www.wearedevelopers.com/jobs/ext/2219504-aws-databricks-data-engineer). Every page supports `.md` or `Accept: text/markdown`. Links point to the HTML versions so they work for humans too. Agent guide: [/agents.md](https://www.wearedevelopers.com/agents.md).

---

# AWS Databricks Data Engineer

- **Company:** Lightning Minds Inc.
- **Location:** United States (Remote available)
- **Experience:** Expert
- **Contract:** Permanent contract
- **Skills:** Amazon Web Services, Amazon S3, Automation of Tests, Computer Programming, Continuous Integration, Information Engineering, Data Governance, Data Infrastructure, Github, Identity and Access Management, Python (Programming Language), Machine Learning, Medicaid Management Information Systems (MMIS), SQL Databases, Data Streaming, Data Storage Technologies, Apache Spark, Gitlab, Data Lakes, Pyspark, Data Lakehouse, Functional Programming, Data Pipelines, Databricks
- **Published:** August 25, 2026
- **Apply:** https://www.dice.com/job-detail/0f0e4c5b-6e0f-46c9-bdc7-b502cfffa7cf

## About the Role

We are seeking an experienced AWS Databricks Data Engineer to design, build, and maintain scalable data pipelines and cloudnative data infrastructure. The ideal candidate will have strong expertise in Apache Spark, PySpark, Python, SQL, Delta Lake, and AWS services, with proven experience in production Databricks environments., * 3 5 years of handson data engineering experience.

 * Strong productionlevel experience with Databricks.
 * Advanced programming skills in Python and PySpark.
 * Advanced SQL expertise.
 * Deep understanding of Apache Spark, Delta Lake, and Data Lakehouse architecture.
 * Experience with Delta Live Tables, Unity Catalog, and Databricks Workflows.
 * Familiarity with AWS cloud services for data storage and management.

## Description

* Design and implement batch and streaming data pipelines using PySpark, Delta Lake, and Delta Live Tables (DLT).
 * Develop scalable ingestion and transformation solutions within Databricks.
 * Build integrations with AWS services such as S3, IAM, and Lambda.
 * Configure and manage data governance, security, and access controls via Databricks Unity Catalog.
 * Monitor and optimize pipeline performance, reliability, and data quality.
 * Automate testing and deployment using CI/CD tools (GitHub/GitLab).
 * Collaborate with data scientists and stakeholders to prepare datasets for analytics and machine learning.
 * Support healthcare payer datasets including claims, provider, member enrollment, and MMIS data.


## Related Videos

- [PySpark - Combining Machine Learning & Big Data](https://www.wearedevelopers.com/videos/44-pyspark-combining-machine-learning-big-data)
- [Innovating Developer Tools with AI: Insights from GitHub Next](https://www.wearedevelopers.com/videos/1268-innovating-developer-tools-with-ai-insights-from-github-next)
- [WeAreDevelopers LIVE - Modern DevOps for IoT Devices and More](https://www.wearedevelopers.com/videos/1805-wearedevelopers-live-modern-devops-for-iot-devices-and-more)
- [Empowering Retail Through Applied Machine Learning](https://www.wearedevelopers.com/videos/976-empowering-retail-through-applied-machine-learning)
- [Data Science in Retail](https://www.wearedevelopers.com/videos/586-data-science-in-retail)
- [Enjoying SQL data pipelines with dbt](https://www.wearedevelopers.com/videos/823-enjoying-sql-data-pipelines-with-dbt)

## Related Articles

- [Top Big Data Technologies That You Need to Know](https://www.wearedevelopers.com/magazine/108-top-big-data-technologies-that-you-need-to-know)
- [Making Data Warehouses Fast: A Developer’s Story](https://www.wearedevelopers.com/magazine/107-making-data-warehouses-fast-a-developer-s-story)
- [Data Engineer Salary UK](https://www.wearedevelopers.com/magazine/253-data-engineer-salary-uk)
- [Highest Paying Tech Companies for Developers](https://www.wearedevelopers.com/magazine/220-highest-paying-tech-companies-for-developers)
- [How to Become an AI Engineer](https://www.wearedevelopers.com/magazine/331-how-to-become-an-ai-engineer)
- [Data Analyst Salary in the UK](https://www.wearedevelopers.com/magazine/278-data-analyst-salary-in-the-uk)