AWS Databricks Data Engineer
Lightning Minds Inc.
United States
about 2 months ago
- Discuss this with your agent
- Open in Claude
- Open in ChatGPT
Apply on www.dice.com
Prepare application
- Draft this with your agent
- Open in Claude
- Open in ChatGPT
Role details
Contract type
Permanent contract
Employment type
Full-time (> 32 hours)
Experience level
Expert
Experience required
5 years minimum
Working hours
Regular working hours
Job source
Tech stack
Amazon Web Services
Amazon S3
Automation of Tests
Computer Programming
Continuous Integration
Information Engineering
Data Governance
Data Infrastructure
Github
Identity and Access Management
Python (Programming Language)
Machine Learning
+12 more
Medicaid Management Information Systems (MMIS)
SQL Databases
Data Streaming
Data Storage Technologies
Apache Spark
Gitlab
Data Lakes
Pyspark
Data Lakehouse
Functional Programming
Data Pipelines
Databricks
Job description
- Design and implement batch and streaming data pipelines using PySpark, Delta Lake, and Delta Live Tables (DLT).
- Develop scalable ingestion and transformation solutions within Databricks.
- Build integrations with AWS services such as S3, IAM, and Lambda.
- Configure and manage data governance, security, and access controls via Databricks Unity Catalog.
- Monitor and optimize pipeline performance, reliability, and data quality.
- Automate testing and deployment using CI/CD tools (GitHub/GitLab).
- Collaborate with data scientists and stakeholders to prepare datasets for analytics and machine learning.
- Support healthcare payer datasets including claims, provider, member enrollment, and MMIS data.
Requirements
We are seeking an experienced AWS Databricks Data Engineer to design, build, and maintain scalable data pipelines and cloudnative data infrastructure. The ideal candidate will have strong expertise in Apache Spark, PySpark, Python, SQL, Delta Lake, and AWS services, with proven experience in production Databricks environments., * 3 5 years of handson data engineering experience.
- Strong productionlevel experience with Databricks.
- Advanced programming skills in Python and PySpark.
- Advanced SQL expertise.
- Deep understanding of Apache Spark, Delta Lake, and Data Lakehouse architecture.
- Experience with Delta Live Tables, Unity Catalog, and Databricks Workflows.
- Familiarity with AWS cloud services for data storage and management.
Apply for this position
This job is hosted externally. Click below to view the full posting and apply.
Apply on www.dice.com
Prepare application
- Draft this with your agent
- Open in Claude
- Open in ChatGPT
Good distractions
Loading talks and stories from around this roleβ¦