Azure Databricks Data Engineer
SFE PARTNERS LLC
St. Louis, MO, United States
5 days ago
- Discuss this with your agent
- Open in Claude
- Open in ChatGPT
Apply on www.careerjet.com
Prepare application
- Draft this with your agent
- Open in Claude
- Open in ChatGPT
Role details
Contract type
Permanent contract
Employment type
Full-time (> 32 hours)
Working hours
Regular working hours
Job source
Tech stack
Application Frameworks
Automation of Tests
Microsoft Azure
Code Review
Continuous Integration
Information Engineering
Data Governance
Data Structures
Software Debugging
DevOps
Apache Hive
Metadata
+20 more
Performance Tuning
Query Optimization
Power BI
Azure Data Lake
Enterprise Data Management
Data Processing
Sql Optimization
Apache Spark
Git
Ceridian Dayforce
Data Lakes
Pyspark
Integration Tests
People Soft
Deployment Automation
Cloud Integration
Software Version Control
Data Pipelines
Workday
Databricks
Job description
- Designing and developing scalable batch data pipelines in Azure Databricks using PySpark and Spark SQL.
- Building and maintaining Bronze, Silver and Gold data structures, including certified fact and dimension data products.
- Translating business metric definitions, calculation rules, filters, granularity, history and refresh requirements into technical designs.
- Creating reusable HR-domain transformations covering areas such as retention, hiring, transfers, promotions, reviews, hours and scheduling.
- Implementing data-quality checks, reconciliation, exception handling, auditability, lineage and technical documentation.
- Optimizing Spark workloads, Delta tables, joins, partitions, file sizes and cluster usage for performance and cost.
- Applying enterprise security/governance standards, including Unity Catalog, permissions, sensitive-data handling and row-level access patterns.
- Coordinating with ingestion, data governance, modeling, DevOps and Power BI teams to establish clear data contracts and acceptance criteria.
- Contributing to CI/CD, source control, automated testing, deployment, operational monitoring and production-readiness activities.
- Participating in requirement workshops and communicating technical decisions to business, HR, analytics and engineering stakeholders.
- Supporting knowledge transfer and mentoring engineers/analysts.
Requirements
- Significant Data Engineering experience with substantial Azure Databricks development.
- Azure Databricks production-grade batch pipelines and Lakehouse solutions.
- PySpark / Spark SQL advanced transformations, joins, window functions, reusable frameworks, debugging and performance tuning.
- Delta Lake table design, schema evolution, merge/upsert, optimization, data reliability and history management.
- Medallion Architecture Bronze/source-aligned, Silver/conformed and Gold/consumption-ready layers.
- Data Modeling fact/dimension design, conformed dimensions, SCDs, aggregates and semantic-consumption considerations.
- Data Quality & Testing profiling, rules, reconciliation, unit/integration testing, controls and defect analysis.
- Azure ecosystem Azure Data Lake Storage plus Azure integration, identity, monitoring and DevOps services.
- DevOps Git-based development, code reviews, CI/CD, environment promotion, release controls and automated deployment.
- Security & Governance least-privilege access, catalog, lineage, classification and row-level security.
- Advanced SQL and query optimization for large analytical datasets.
- Hands-on exposure to Databricks Lakeflow Spark Declarative Pipelines / Delta Live Tables (DLT) and ability to adopt evolving Databricks-native pipeline standards.
- Unity Catalog experience.
- Familiarity with enterprise data-governance/data-quality tools such as Informatica.
- Experience with HR data, preferably from Workday, Dayforce, PeopleSoft, recruiting, workforce management or talent systems.
- Power BI consumption patterns, star schemas, refresh requirements, aggregates and performance.
- Experience building accelerators, reusable engineering templates, metadata-driven pipelines or automated test frameworks.
- Databricks Data Engineer certification or equivalent cloud/data engineering credential.
- Experience working in a blended onshore/offshore Agile delivery model.
Apply for this position
This job is hosted externally. Click below to view the full posting and apply.
Apply on www.careerjet.com
Prepare application
- Draft this with your agent
- Open in Claude
- Open in ChatGPT
Good distractions
Loading talks and stories from around this roleβ¦