Sr Data Engineer (Azure Databricks)
Insight Global
Woonsocket, RI, United States
14 days ago
- Discuss this with your agent
- Open in Claude
- Open in ChatGPT
Apply on www.techcareers.com
Prepare application
- Draft this with your agent
- Open in Claude
- Open in ChatGPT
Role details
Contract type
Permanent contract
Employment type
Full-time (> 32 hours)
Experience level
Expert
Experience required
5 years minimum
Working hours
Regular working hours
Job source
Tech stack
Big Data
Continuous Integration
Information Engineering
Data Warehousing
JSON
Python (Programming Language)
Operational Databases
Performance Tuning
Standard Sql
Azure Data Lake
SQL Databases
YAML
+7 more
Delivery Pipeline
Infrastructure as Code (IaC)
Build Management
Data Lakes
Pyspark
Data Pipelines
Databricks
Job description
- Design, develop, and deploy Azure Databricks data pipelines using PySpark and Delta Live Tables
- Modernize legacy on-prem SQL pipelines into scalable cloud-native Databricks solutions
- Build configuration-driven data pipelines using YAML, JSON, and reusable framework patterns
- Develop and maintain CI/CD deployment processes for data engineering workflows
- Implement and support Medallion Architecture pipelines across Bronze, Silver, and Gold layers
- Partner with upstream and downstream teams to ensure data quality, reliability, and scalability
- Troubleshoot and resolve production pipeline issues across Databricks and legacy SQL environments
- Enhance monitoring, observability, and operational support processes
- Support Medicare and payer-focused initiatives involving member, claims, and operational datasets
- Follow established architecture standards and engineering best practices to ensure platform consistency and maintainability
Requirements
- 5-7+ years of Data Engineering experience building large-scale data platforms
- Strong hands-on experience with Azure Databricks, Delta Lake, and Azure Data Lake Storage (ADLS)
- Advanced PySpark development experience, including DataFrames, transformations, joins, aggregations, and performance tuning
- Experience developing and maintaining Delta Live Tables (DLT) pipelines
- Strong SQL background with experience supporting legacy data warehouse environments
- Experience migrating SQL-based pipelines to modern Python/PySpark architectures
- Experience with Infrastructure as Code (IaC) and Pipeline as Code methodologies
- Proficiency with YAML, JSON, and configuration-driven development patterns
- Experience automating CI/CD deployments using DevOps pipelines
- Strong understanding of Medallion Architecture (Bronze, Silver, Gold)
- Experience building ingestion, transformation, and curation pipelines across Bronze, Silver, and Gold layers
- Experience troubleshooting, monitoring, and supporting production data pipelines
- Familiarity with Databricks Asset Bundles, Unity Catalog, and Delta Lake preferred
- Healthcare industry experience preferred
- Medicare, payer, member enrollment, or claims data experience strongly preferred Understanding of healthcare data quality, PHI/PII handling, and regulatory requirements preferred
Apply for this position
This job is hosted externally. Click below to view the full posting and apply.
Apply on www.techcareers.com
Prepare application
- Draft this with your agent
- Open in Claude
- Open in ChatGPT
Good distractions
Loading talks and stories from around this roleβ¦