> Markdown version of [/jobs/ext/2700753-principal-data-engineer](https://www.wearedevelopers.com/jobs/ext/2700753-principal-data-engineer). Every page supports `.md` or `Accept: text/markdown`. Links point to the HTML versions so they work for humans too. Agent guide: [/agents.md](https://www.wearedevelopers.com/agents.md). --- # Principal Data Engineer - **Company:** KPMG International Cooperative - **Location:** Birmingham, UK - **Experience:** Expert - **Contract:** Permanent contract - **Skills:** Agile Methodology, Artificial Intelligence, Apache HTTP Server, Microsoft Azure, Encodings, Continuous Integration, Information Engineering, Data Governance, Extract Transform Load (ETL), Data Transformation, Data Systems, Decision Support Systems, DevOps, Github, Python (Programming Language), Performance Tuning, Scrum Methodology, Role-Based Access Control, Power BI, SQL Databases, Data Streaming, TypeScript, Data Processing, Azure Data Factory, GitHub Copilot, Delivery Pipeline, Git, Event Driven Architecture, Data Lakes, Pyspark, Storage Technologies, Information Technology, Data Lineage, Maintaining Code, Bicep, Enterprise Integration, Apache Kafka, Data Management, Virtual Agents, Terraform, Stream Processing, Data Pipelines, Databricks - **Published:** September 4, 2026 - **Apply:** https://performancemanager5.successfactors.eu/career?career_ns=job_application&company=KPMGHUKPROD&career_job_req_id=109389&selected_lang=en_GB ## About the Role * Experience in data engineering disciplines, with advanced, hands-on expertise in Databricks platform including PySpark framework, Delta Lake storage format, and Unity Catalog governance * Extensive experience in designing advanced data models, including medallion architectures, data lakes, and enterprise data warehouses * Expert-level proficiency in SQL for complex data transformation, optimization, and analytical workload development * Advanced proficiency in Python for data engineering, pipeline automation, and integration development * Deep expertise in PySpark for data processing and transformation * Proven mastery of cloud platforms with strong preference for Microsoft Azure * Expertise in ETL/ELT development and data pipeline orchestration (e.g. Databricks Workflows, DLT, ADF) * Extensive experience with Git version control, GitHub collaboration workflows, and modern CI/CD practices including automated testing frameworks and deployment pipeline automation for data engineering workloads * Hands-on expertise with Agile software development methodologies (Scrum, Kanban) and DevOps practices applied to data engineering contexts * Demonstrated capability to balance technical execution with people management responsibilities and product accountability * Proven analytical thinking and advanced problem-solving skills with demonstrated ability to tackle complex, enterprise-scale technical challenges * Strong decision-making capabilities with proven ability to evaluate technical trade-offs and make pragmatic choices under pressure and ambiguity * Strong track record in managing and coaching engineering teams (minimum 3+ years in Tech Lead roles) * Experience with AI-native software delivery and specification-driven development, including the use of AI-assisted engineering tools, automated quality assurance, modern testing practices, and highly automated CI/CD workflows within enterprise-scale environments Desirable * Experience implementing data governance frameworks, data lineage tracking, and role-based access control (RBAC) using tools like Apache Atlas, Microsoft Purview * Experience building event-driven architectures and real-time data streaming pipelines using tools like Apache Kafka, AWS Kinesis, or Azure Event Hubs * Familiarity with agentic AI frameworks and architectures such as Microsoft Agent Framework or equivalent frameworks in Python, TypeScript, or other languages * Experience architecting data solutions for scalability, performance optimization, security hardening, and operational excellence in production environments * Knowledge of Infrastructure as Code (IaC) frameworks including Terraform or Azure Bicep for data platform provisioning and management * Background in professional services, consulting, or client-facing technical delivery environments * Active contribution to open-source data engineering projects or participation in data engineering and cloud communities Qualifications (optional): * Relevant associate-level or expert-level professional certifications in Microsoft Azure (e.g., Azure Data Engineer Associate, Azure Solutions Architect Expert), Databricks (e.g., Databricks Certified Data Engineer Professional), or modern development frameworks * Bachelor's degree in Computer Science, Data Science, Engineering, or related technical discipline (or equivalent practical experience) ## Description The Principal Data Engineer will be accountable for shaping the technical direction for data engineering initiatives and services within Technology & Solutions. This role combines deep technical expertise in Databricks and cloud data engineering with proven team leadership capabilities and a passion for driving innovation. The successful candidate will be accountable for delivering enterprise-scale data engineering solutions, championing modern engineering practices including AI-assisted development, and fostering a high-performing team culture focused on excellence, collaboration, and continuous growth. Description of the role: Data Engineering & Delivery Excellence * Design and deliver enterprise-scale data solutions on cloud platforms (Azure preferred), leveraging Databricks as the core data engineering platform * Build and optimize sophisticated ETL/ELT data pipelines using PySpark, SQL, and Python, orchestrating complex data workflows through Databricks Workflows, Delta Live Tables, Azure Data Factory * Implement and maintain Delta Lake storage architectures and Unity Catalog governance frameworks, ensuring data quality, security, and compliance across the data estate * Design data solutions for scalability, performance, resilience, and operational excellence, embedding enterprise-grade standards from inception through production deployment * Lead technical discovery and requirements gathering with senior stakeholders, translating business needs into actionable data platform strategies and technical roadmaps * Integrate data platforms with business intelligence and analytics tools including Databricks AI/BI and Power BI to enable self-service analytics and data-driven decision making Engineering Standards & Modern Data Practices * Establish, evolve, and enforce data engineering standards, coding practices, and quality gates that enable safe, scalable, and maintainable data platform delivery * Champion modern software engineering practices within data engineering contexts, including Git version control workflows, automated testing frameworks, and CI/CD deployment pipelines for data workloads * Drive comprehensive observability, monitoring, and alerting for data pipelines and platforms, ensuring operational readiness and rapid incident response * Promote AI-augmented data engineering practices, leveraging GitHub Copilot, Claude Code, and other approved AI coding assistants to enhance productivity while maintaining code quality and standards Technical Leadership & Team Development * Provide technical direction and engineering accountability for data engineering initiatives, leading teams through hands-on contribution and solving key business challenges * Coach and mentor data engineers at various career levels, fostering a culture of technical excellence, continuous learning, and innovation adoption * Manage delivery of data engineering projects using Agile methodologies (Scrum, Kanban), balancing technical execution with people leadership and stakeholder management * Foster collaboration across engineering, architecture, and product teams, breaking down silos and promoting knowledge sharing across the organization * Represent data engineering in portfolio planning discussions, technical governance forums, and architectural review boards ## Related Videos - [Innovating Developer Tools with AI: Insights from GitHub Next](https://www.wearedevelopers.com/videos/1268-innovating-developer-tools-with-ai-insights-from-github-next) - [Back(end) to the Future: Embracing the continuous Evolution of Infrastructure and Code](https://www.wearedevelopers.com/videos/440-back-end-to-the-future-embracing-the-continuous-evolution-of-infrastructure-and-code) - [How a Small Team Shrank a Microsoft Monorepo by 94%](https://www.wearedevelopers.com/videos/1236-how-a-small-team-shrank-a-microsoft-monorepo-by-94) - [From Messy Queries to Scalable Systems - How Data Engineering actually works](https://www.wearedevelopers.com/videos/100203-from-messy-queries-to-scalable-systems-how-data-engineering-actually-works) - [Bringing AI Model Testing and Prompt Management to Your Codebase with GitHub Models](https://www.wearedevelopers.com/videos/1536-bringing-ai-model-testing-and-prompt-management-to-your-codebase-with-github-models) - [Modern Data Architectures need Software Engineering](https://www.wearedevelopers.com/videos/1030-modern-data-architectures-need-software-engineering) ## Related Articles - [Data Engineer Salary UK](https://www.wearedevelopers.com/magazine/253-data-engineer-salary-uk) - [Data Analyst Salary in the UK](https://www.wearedevelopers.com/magazine/278-data-analyst-salary-in-the-uk) - [How to Become an AI Engineer](https://www.wearedevelopers.com/magazine/331-how-to-become-an-ai-engineer) - [Coffee with Developers - Maria Apazoglou - Making AI understandable for all in production](https://www.wearedevelopers.com/magazine/475-coffee-with-developers-maria-apazoglou-making-ai-understandable-for-all-in-production) - [Highest Paying Tech Companies for Developers](https://www.wearedevelopers.com/magazine/220-highest-paying-tech-companies-for-developers) - [What Are The Top Skills Required For Azure Developers?](https://www.wearedevelopers.com/magazine/77-what-are-the-top-skills-required-for-azure-developers)