> Markdown version of [/jobs/ext/456026-sr-data-engineer-data-intelligence](https://www.wearedevelopers.com/jobs/ext/456026-sr-data-engineer-data-intelligence). Every page supports `.md` or `Accept: text/markdown`. Links point to the HTML versions so they work for humans too. Agent guide: [/agents.md](https://www.wearedevelopers.com/agents.md). --- # Sr Data Engineer - Data & Intelligence - **Company:** Sumeru INC - **Location:** Frisco, TX, United States - **Experience:** Expert - **Contract:** Permanent contract - **Skills:** Microsoft Azure, Big Data, Cloud Storage, Data Architecture, Data Validation, Data Transformation, Data Warehousing, Database Queries, DevOps, Dimensional Modeling, Github, Python (Programming Language), Key Management, Log Analysis, SQL Azure, Query Optimization, Role-Based Access Control, Power BI, Azure Active Directory, Azure Data Lake, SQL Databases, Data Streaming, Azure Data Factory, Cloud Monitoring, Delivery Pipeline, Git, Microsoft Fabric, Data Lakes, Pyspark, Semi-structured Data, Git Flow, Star Schema, Data Management, Data Pipelines, Databricks, Programming Languages - **Published:** June 4, 2026 - **Apply:** https://www.careerjet.com/jobad/usba526cc4418df344936a031821c94469 ## About the Role Experience building and optimizing big data pipelines using Azure Data Factory, PySpark, and SQL across structured and semi-structured data sets Hands-on experience implementing Medallion Architecture (Bronze/Silver/Gold) Experience with Delta Lake - ACID transactions, incremental loading, schema evolution, partitioning strategies Experience performing root cause analysis on pipeline failures and data quality issues to resolve SLA breaches and identify platform improvement opportunities Azure Foundational Services : Working knowledge of: Azure Data Factory (ADF), ADLS Gen2, Azure SQL Database, Azure Blob Storage, Azure Key Vault, Azure Monitor / Log Analytics, Azure Event Hubs, Microsoft Fabric Lakehouse, Azure Active Directory / Entra ID (RBAC, Service Principals) Programming Languages: Proficiency in Python and PySpark for data transformation, pipeline automation, and large-scale distributed processing; strong SQL skills including window functions, CTEs, and query optimization across relational and lakehouse engines Data Architecture: Solid understanding of Medallion Architecture, dimensional modeling (Star Schema, SCD Types 1/2/3), and the trade-offs between lakehouse, data warehouse, and data lake patterns Pipeline Engineering: Ability to build robust ADF pipelines with ForEach, Lookup, Copy Activity, and Data Flows; incremental loading via watermark or CDC; error handling, retry logic, and dead-letter patterns Data Quality Experience: Experience implementing SLA-based data quality checks (freshness, completeness, row count), monitoring via Azure Monitor and ADF diagnostic logs, and defining data quality agreements with business stakeholders. DevOps for Data: Experience with Git-based workflows, ADF Git integration, CI/CD pipeline promotion across Dev/Test/Prod using Azure DevOps or GitHub Actions Reporting Layer Awareness: Understanding of how Gold-layer data feeds Power BI - DirectQuery vs. Import mode trade-offs, dataset refresh patterns, and semantic model collaboration with BI teams Ability to manage work across multiple concurrent pipeline projects, prioritize by business impact, and communicate status clearly to technical and non-technical stakeholders Good to have skills: Experience with Microsoft Fabric (Lakehouse, Notebooks, OneLake, Fabric Pipelines) - active migration or greenfield project Experience with real-time / streaming workloads using Azure Event Hubs or Structured Streaming in PySpark Experience delivering data platforms for executive-level reporting via Power BI semantic ## Description This role builds, and maintains scalable data pipelines and lakehouse infrastructure on Microsoft Azure to support efficient extraction, transformation, and loading of data across batch and real-time workloads. It involves implementing and managing the Medallion Architecture (Bronze Silver Gold) using Azure Data Factory, Databricks-PySpark, and Azure SQL Database and Databricks unity catalogue. The role requires ensuring SLA-adherent data quality standards. Success is measured by pipeline reliability, data freshness SLA compliance, and the quality of Gold-layer datasets powering Power BI executive dashboards. The work supports organizational decision-making by delivering trusted, well-governed data to business executives and analytics consumers. ## Related Videos - [From Messy Queries to Scalable Systems - How Data Engineering actually works](https://www.wearedevelopers.com/videos/100203-from-messy-queries-to-scalable-systems-how-data-engineering-actually-works) - [Innovating Developer Tools with AI: Insights from GitHub Next](https://www.wearedevelopers.com/videos/1268-innovating-developer-tools-with-ai-insights-from-github-next) - [From DevOps to Scaled DevOps: How We’re Rebuilding Continuous Delivery as a Platform](https://www.wearedevelopers.com/videos/100018-from-devops-to-scaled-devops-how-we-re-rebuilding-continuous-delivery-as-a-platform) - [How a Small Team Shrank a Microsoft Monorepo by 94%](https://www.wearedevelopers.com/videos/1236-how-a-small-team-shrank-a-microsoft-monorepo-by-94) - [Alibaba Big Data and Machine Learning Technology](https://www.wearedevelopers.com/videos/37-alibaba-big-data-and-machine-learning-technology) - [Bringing AI Model Testing and Prompt Management to Your Codebase with GitHub Models](https://www.wearedevelopers.com/videos/1536-bringing-ai-model-testing-and-prompt-management-to-your-codebase-with-github-models) ## Related Articles - [Making Data Warehouses Fast: A Developer’s Story](https://www.wearedevelopers.com/magazine/107-making-data-warehouses-fast-a-developer-s-story) - [What Are The Top Skills Required For Azure Developers?](https://www.wearedevelopers.com/magazine/77-what-are-the-top-skills-required-for-azure-developers) - [Highest Paying Tech Companies for Developers](https://www.wearedevelopers.com/magazine/220-highest-paying-tech-companies-for-developers) - [Data Engineer Salary UK](https://www.wearedevelopers.com/magazine/253-data-engineer-salary-uk) - [Top Big Data Technologies That You Need to Know](https://www.wearedevelopers.com/magazine/108-top-big-data-technologies-that-you-need-to-know) - [Coffee with Developers - Maria Apazoglou - Making AI understandable for all in production](https://www.wearedevelopers.com/magazine/475-coffee-with-developers-maria-apazoglou-making-ai-understandable-for-all-in-production)