> Markdown version of [/jobs/ext/3671601-azure-databricks-data-engineer](https://www.wearedevelopers.com/jobs/ext/3671601-azure-databricks-data-engineer). Every page supports `.md` or `Accept: text/markdown`. Links point to the HTML versions so they work for humans too. Agent guide: [/agents.md](https://www.wearedevelopers.com/agents.md).

---

# Azure Databricks Data Engineer

- **Company:** SFE PARTNERS LLC
- **Location:** St. Louis, MO, United States
- **Contract:** Permanent contract
- **Skills:** Application Frameworks, Automation of Tests, Microsoft Azure, Code Review, Continuous Integration, Information Engineering, Data Governance, Data Structures, Software Debugging, DevOps, Apache Hive, Metadata, Performance Tuning, Query Optimization, Power BI, Azure Data Lake, Enterprise Data Management, Data Processing, Sql Optimization, Apache Spark, Git, Ceridian Dayforce, Data Lakes, Pyspark, Integration Tests, People Soft, Deployment Automation, Cloud Integration, Software Version Control, Data Pipelines, Workday, Databricks
- **Published:** October 10, 2026
- **Apply:** https://www.careerjet.com/jobad/usf751526d474f482fc74502aaa270950c

## About the Role

* Significant Data Engineering experience with substantial Azure Databricks development.
 * Azure Databricks production-grade batch pipelines and Lakehouse solutions.
 * PySpark / Spark SQL advanced transformations, joins, window functions, reusable frameworks, debugging and performance tuning.
 * Delta Lake table design, schema evolution, merge/upsert, optimization, data reliability and history management.
 * Medallion Architecture Bronze/source-aligned, Silver/conformed and Gold/consumption-ready layers.
 * Data Modeling fact/dimension design, conformed dimensions, SCDs, aggregates and semantic-consumption considerations.
 * Data Quality & Testing profiling, rules, reconciliation, unit/integration testing, controls and defect analysis.
 * Azure ecosystem Azure Data Lake Storage plus Azure integration, identity, monitoring and DevOps services.
 * DevOps Git-based development, code reviews, CI/CD, environment promotion, release controls and automated deployment.
 * Security & Governance least-privilege access, catalog, lineage, classification and row-level security.
 * Advanced SQL and query optimization for large analytical datasets.
 * Hands-on exposure to Databricks Lakeflow Spark Declarative Pipelines / Delta Live Tables (DLT) and ability to adopt evolving Databricks-native pipeline standards.
 * Unity Catalog experience.
 * Familiarity with enterprise data-governance/data-quality tools such as Informatica.
 * Experience with HR data, preferably from Workday, Dayforce, PeopleSoft, recruiting, workforce management or talent systems.
 * Power BI consumption patterns, star schemas, refresh requirements, aggregates and performance.
 * Experience building accelerators, reusable engineering templates, metadata-driven pipelines or automated test frameworks.
 * Databricks Data Engineer certification or equivalent cloud/data engineering credential.
 * Experience working in a blended onshore/offshore Agile delivery model.

## Description

* Designing and developing scalable batch data pipelines in Azure Databricks using PySpark and Spark SQL.
 * Building and maintaining Bronze, Silver and Gold data structures, including certified fact and dimension data products.
 * Translating business metric definitions, calculation rules, filters, granularity, history and refresh requirements into technical designs.
 * Creating reusable HR-domain transformations covering areas such as retention, hiring, transfers, promotions, reviews, hours and scheduling.
 * Implementing data-quality checks, reconciliation, exception handling, auditability, lineage and technical documentation.
 * Optimizing Spark workloads, Delta tables, joins, partitions, file sizes and cluster usage for performance and cost.
 * Applying enterprise security/governance standards, including Unity Catalog, permissions, sensitive-data handling and row-level access patterns.
 * Coordinating with ingestion, data governance, modeling, DevOps and Power BI teams to establish clear data contracts and acceptance criteria.
 * Contributing to CI/CD, source control, automated testing, deployment, operational monitoring and production-readiness activities.
 * Participating in requirement workshops and communicating technical decisions to business, HR, analytics and engineering stakeholders.
 * Supporting knowledge transfer and mentoring engineers/analysts.