Senior Azure Databricks & Microsoft Fabric Data Engineer

Donato Technologies, Inc
Houston, TX, United States
3 days ago
Apply on www.careerjet.com
Prepare application

Role details

Contract type
Permanent contract
Employment type
Full-time (> 32 hours)
Experience level
Expert
Working hours
Regular working hours

Tech stack

Data Analysis IBM System I Microsoft Azure Big Data Code Review Computer Programming Continuous Integration Data Architecture Information Engineering Data Governance Data Integration Data Transformation
+38 more
Data Security Data Systems Data Warehousing IBM DB2 DevOps Dimensional Modeling Github JD Edwards EnterpriseOne Python (Programming Language) Oracle (Applications) Performance Tuning Primavera Release Management Power BI SQL Databases Systems Integration Enterprise Data Management Data Logging Data Processing Enterprise Software Applications Cloud Platform System Data Ingestion Microsoft Power Automate HybridCloud Data Layers Microsoft Fabric Data Lakes Pyspark Infrastructure Automation Frameworks Deployment Automation Bicep Data Management Restful APIs Terraform Software Version Control Data Pipelines Legacy Systems Databricks

Job description

We are seeking an experienced Senior Data Engineer with strong hands-on expertise in Azure Databricks, Microsoft Fabric, Python, SQL, and modern cloud data engineering technologies. The ideal candidate will be responsible for designing, developing, implementing, and maintaining scalable enterprise data platforms and production-grade data pipelines. The candidate should have strong experience building modern Lakehouse architectures, implementing data governance, integrating data from enterprise applications, and developing automated CI/CD processes. This role requires someone who can work across Azure Databricks and Microsoft Fabric environments to deliver secure, reliable, and high-performing data solutions., 1. Azure Databricks & Data Engineering

  • Design, develop, and maintain scalable data engineering solutions using Azure Databricks.
  • Implement data governance and access controls using Unity Catalog.
  • Develop and optimize data pipelines using Delta Lake, PySpark, and Lakeflow / Delta Live Tables (DLT).
  • Build and manage automated deployment processes using Databricks Asset Bundles.
  • Implement data quality, monitoring, and performance optimization across data pipelines.
  • Work with large datasets and develop reliable batch and incremental data processing solutions.
  1. Microsoft Fabric & Lakehouse Architecture * Design and implement modern Lakehouse solutions using Microsoft Fabric. * Work with OneLake for centralized enterprise data storage and management. * Develop and manage Data Pipelines and Dataflows for data ingestion and transformation. * Understand and implement Fabric networking capabilities, including Managed Private Endpoints and VNet Data Gateways. * Integrate Microsoft Fabric with Azure Databricks and other enterprise data platforms. * Ensure secure, scalable, and efficient data access across cloud environments.

  2. SQL, Python & Data Modeling * Develop production-ready data pipelines using strong Python and SQL programming skills. * Build data transformation workflows using PySpark and SQL. * Apply dimensional data modeling techniques to support enterprise reporting and analytics. * Implement Medallion Architecture, including Bronze, Silver, and Gold data layers. * Develop reusable data transformation frameworks and maintain high-quality data standards. * Experience with dbt for data transformation and modeling is highly preferred.

  3. Enterprise Data Integration * Design and implement data ingestion solutions from REST APIs, enterprise applications, and legacy systems. * Integrate data from ERP and enterprise platforms such as:

  • JD Edwards (JDE)
  • IBM DB2 / IBM i
  • Oracle Primavera P6
  • Other ERP and legacy enterprise systems.
  • Develop integrations using Azure services such as Azure Durable Functions, Logic Apps, and Service Bus.
  • Build reliable data ingestion processes with appropriate error handling, logging, retry mechanisms, and monitoring.
  • Work with business and application teams to understand source systems and data requirements.
  1. DevOps, CI/CD & Infrastructure as Code * Develop and maintain automated CI/CD pipelines using Azure DevOps or GitHub Actions. * Implement infrastructure provisioning and configuration using Bicep or Terraform. * Manage deployment of data engineering solutions across development, testing, and production environments. * Work with service principals and Workload Identity Federation for secure authentication. * Implement environment promotion strategies and automated deployment practices. * Follow version control, code review, release management, and DevOps best practices.

  2. Power BI & Semantic Data Layer * Support downstream analytics and reporting solutions using Power BI. * Understand and work with Direct Lake semantic models in Microsoft Fabric. * Support semantic model development, governance, and performance optimization. * Ensure data pipelines deliver reliable, analytics-ready datasets for reporting and business intelligence. * Understand data serving requirements and integration between data platforms and reporting layers. * Exposure to Microsoft Power Platform and Dataverse is an added advantage., The ideal candidate should be a hands-on Senior Data Engineer who has worked extensively on modern Azure-based data platforms. The candidate must be comfortable developing end-to-end data pipelines, integrating enterprise and legacy systems, implementing data governance, and managing automated deployments.

Requirements

  • Strong hands-on experience with Azure Databricks.
  • Expertise in Unity Catalog, Delta Lake, PySpark, and Lakeflow / DLT.
  • Experience with Databricks Asset Bundles and deployment automation.
  • Hands-on experience with Microsoft Fabric, OneLake, Lakehouse, Data Pipelines, and Dataflows.
  • Strong programming skills in Python and SQL.
  • Experience with dimensional modeling and Medallion Architecture.
  • Experience developing enterprise data integration solutions using REST APIs.
  • Knowledge of ERP and legacy systems such as JDE, DB2 / IBM i, and Oracle P6.
  • Experience with Azure Durable Functions, Logic Apps, and Service Bus.
  • Strong understanding of CI/CD using Azure DevOps or GitHub.
  • Experience with Terraform or Bicep for Infrastructure as Code.
  • Understanding of secure cloud authentication using service principals and Workload Identity Federation.
  • Knowledge of Power BI and Direct Lake semantic models.

Preferred / Good-to-Have Skills

  • Experience with dbt for data transformation.
  • Exposure to Power Platform and Dataverse.
  • Experience integrating multiple enterprise data sources.
  • Knowledge of enterprise data governance, security, and compliance.
  • Experience optimizing large-scale data processing workloads.
  • Familiarity with hybrid cloud and enterprise networking configurations., We are looking for someone with practical project experience in both Azure Databricks and Microsoft Fabric, rather than someone with only theoretical knowledge or exposure to these technologies.

About the company

Donato Technologies Inc. is a trusted IT staffing, consulting, and software development partner headquartered in Dallas, Texas. We support clients across industries by understanding their unique business needs and delivering tailored technology and workforce solutions. Our focus is on connecting the right talent with the right opportunity-ensuring clients receive dependable, skilled professionals and candidates receive meaningful career growth and support. We work closely with small to mid-sized organizations to provide flexible, high-quality services that drive performance, innovation, and long-term success.

Apply for this position

This job is hosted externally. Click below to view the full posting and apply.

Apply on www.careerjet.com
Prepare application

Good distractions

Loading talks and stories from around this role…