> Markdown version of [/jobs/ext/2551906-aws-databricks-data-engineer](https://www.wearedevelopers.com/jobs/ext/2551906-aws-databricks-data-engineer). Every page supports `.md` or `Accept: text/markdown`. Links point to the HTML versions so they work for humans too. Agent guide: [/agents.md](https://www.wearedevelopers.com/agents.md).

---

# AWS Databricks Data Engineer

- **Company:** Palni Inc
- **Location:** United States (Remote available)
- **Experience:** Experienced
- **Contract:** Temporary to permanent
- **Skills:** Airflow, Amazon Web Services, Amazon S3, Cloud Computing, Cloud Storage, Computer Programming, Continuous Integration, Information Engineering, Data Files, Data Governance, Github, Identity and Access Management, Python (Programming Language), Medicaid Management Information Systems (MMIS), SQL Databases, Data Streaming, Apache Spark, Gitlab, Build Management, Data Lakes, Pyspark, AWS Glue, Data Lakehouse, Data Pipelines, Serverless Computing, Databricks
- **Published:** August 26, 2026
- **Apply:** https://www.dice.com/job-detail/fd76e1fe-eda4-4375-8238-0ceac2c5caf7

## About the Role

* Industry: Healthcare Payer industry experience. Have worked on MMIS data sets - claims, provider, member enrollment and similar data sets
 * Experience: 3-5 years of hands-on data engineering experience, specifically on Databricks.
 * Programming: High proficiency in Python (specifically PySpark) and advanced SQL.
 * Big Data & Cloud: Strong understanding of Apache Spark, Data Lakehouse architecture, and working within a production AWS environment.
 * Databricks Ecosystem: Familiarity with the Databricks platform ecosystem, including notebooks, Delta Lake, and Unity Catalog.

## Description

6months Contract/Contract to Hire
Must be authorized to work in the U.S. without sponsorship. No EAD''s
An AWS Databricks Data Engineer to design, build, and maintains scalable data pipelines and lakehouse architectures using Apache Spark, Python, and SQL. Optimize cloud storage, ensure data quality, and integrate seamlessly with other AWS services

Core Responsibilities

 * Pipeline Development: Design and build batch and streaming data pipelines using PySpark, Delta Lake, Autoloader and Delta Live Tables (DLT) to ingest data sets into Databricks.
 * AWS Integration: Build cloud-native data solutions leveraging AWS services (e.g., S3 for storage, IAM for access management, and AWS Glue or Lambda for serverless tasks
 * Data Governance & Security: Configure and manage access controls using Databricks Unity Catalog to ensure compliance and monitor lineage
 * Orchestration & CI/CD: Automate pipeline deployments using Databricks Workflows, Apache Airflow, and CI/CD tools (e.g., GitHub, GitLab).
 * Cross-Functional Collaboration: Partner closely with stakeholders to build robust feature stores and prepare datasets for various consumption needs


## Related Videos

- [From Messy Queries to Scalable Systems - How Data Engineering actually works](https://www.wearedevelopers.com/videos/100203-from-messy-queries-to-scalable-systems-how-data-engineering-actually-works)
- [Alibaba Big Data and Machine Learning Technology](https://www.wearedevelopers.com/videos/37-alibaba-big-data-and-machine-learning-technology)
- [Innovating Developer Tools with AI: Insights from GitHub Next](https://www.wearedevelopers.com/videos/1268-innovating-developer-tools-with-ai-insights-from-github-next)
- [WeAreDevelopers LIVE - Modern DevOps for IoT Devices and More](https://www.wearedevelopers.com/videos/1805-wearedevelopers-live-modern-devops-for-iot-devices-and-more)
- [Enjoying SQL data pipelines with dbt](https://www.wearedevelopers.com/videos/823-enjoying-sql-data-pipelines-with-dbt)
- [AI Model Management Life Circles: ML Ops For Generative AI Models From Research to Deployment](https://www.wearedevelopers.com/videos/1152-ai-model-management-life-circles-ml-ops-for-generative-ai-models-from-research-to-deployment)

## Related Articles

- [Top Big Data Technologies That You Need to Know](https://www.wearedevelopers.com/magazine/108-top-big-data-technologies-that-you-need-to-know)
- [Making Data Warehouses Fast: A Developer’s Story](https://www.wearedevelopers.com/magazine/107-making-data-warehouses-fast-a-developer-s-story)
- [Highest Paying Tech Companies for Developers](https://www.wearedevelopers.com/magazine/220-highest-paying-tech-companies-for-developers)
- [Data Engineer Salary UK](https://www.wearedevelopers.com/magazine/253-data-engineer-salary-uk)
- [Data Analyst Salary in the UK](https://www.wearedevelopers.com/magazine/278-data-analyst-salary-in-the-uk)
- [Coffee with Developers - Maria Apazoglou - Making AI understandable for all in production](https://www.wearedevelopers.com/magazine/475-coffee-with-developers-maria-apazoglou-making-ai-understandable-for-all-in-production)