> Markdown version of [/jobs/ext/2820431-manager-senior-databricks-data-ai-engineer-technical-lead-databricks-architect](https://www.wearedevelopers.com/jobs/ext/2820431-manager-senior-databricks-data-ai-engineer-technical-lead-databricks-architect). Every page supports `.md` or `Accept: text/markdown`. Links point to the HTML versions so they work for humans too. Agent guide: [/agents.md](https://www.wearedevelopers.com/agents.md). --- # Manager, Senior Databricks Data & AI Engineer/Technical Lead/Databricks Architect - **Company:** Deloitte - **Location:** London, UK - **Experience:** Expert - **Contract:** Permanent contract - **Skills:** Unity 3d, Agile Methodology, Artificial Intelligence, Airflow, Amazon Web Services, Computing Platforms, Automation of Tests, Microsoft Azure, Code Review, Continuous Delivery, Continuous Integration, Data Architecture, Information Engineering, DevOps, Github, Industry Standard Architecture, Python (Programming Language), Machine Learning, Performance Tuning, Search Technologies, Software Construction, Software Deployment, Software Engineering, SQL Databases, Data Streaming, Feature Engineering, Azure Data Factory, Large Language Models, Prompt Engineering, Apache Spark, Generative AI, Git, Data Lakes, Pyspark, Kubernetes, Infrastructure Automation Frameworks, Machine Learning Operations, Virtual Agents, Data Pipelines, Docker, Databricks - **Published:** September 10, 2026 - **Apply:** https://apply.deloitte.co.uk/UKCareers/Login?jobId=25087 ## About the Role You're passionate about mentoring others, making sound architectural decisions, and delivering innovative Data & AI solutions that create measurable business value. You'll have strong technical skills including some or all of the following: * Extensive experience delivering enterprise-scale Data Engineering and AI solutions using the Databricks Data Intelligence Platform. * Proven experience as a solution or platform architect, defining reference architectures, integration patterns, and target-state designs across enterprise Databricks environments. * Deep expertise across Databricks including Delta Lake, Unity Catalog, Spark, PySpark, Databricks SQL, Databricks Workflows, Lakeflow, Lakehouse architecture, MLflow, AI/BI Genie, Mosaic AI, and Agent Bricks. * Proven experience leading engineering teams across both onshore and offshore delivery models. * Strong experience establishing engineering standards, technical governance, and design authority across large programmes. * Expertise in software engineering best practices including Git, CI/CD, automated testing, Infrastructure as Code, and peer code reviews. * Strong experience designing, implementing and optimising large-scale batch and streaming data pipelines. * Advanced Spark performance tuning and optimisation experience across storage, compute, and workload execution. * Experience delivering production Machine Learning solutions using MLflow, feature engineering, model lifecycle management, and MLOps principles. * Experience designing and delivering enterprise AI solutions, with deep expertise in Agentic AI architectures, AI Agents, Retrieval-Augmented Generation (RAG), Vector Search, LLM orchestration, prompt engineering, AI governance and responsible AI practices. * Excellent communication and stakeholder management skills, with the ability to influence technical and non-technical audiences alike. * Experience working across Azure, AWS, or GCP cloud ecosystems. Required * Proven experience leading enterprise-scale Databricks implementation programmes. * Extensive experience acting as a Technical Lead, Engineering Lead, Solution/Platform Architect, or Technical Design Authority, with responsibility for architecture, solution governance, and technical decision making. * Demonstrable experience leading offshore engineering teams. * Strong hands-on expertise in Python, PySpark, and SQL. * Experience designing & delivering enterprise Generative AI & Agentic AI solutions including AI Agents, RAG, Vector Search, LLM Orchestration and AI governance. * Experience designing & delivering production-grade Machine Learning solutions. * Proven experience conducting solution design reviews, architecture governance, and code reviews. * Strong understanding of modern software engineering and DevOps practices. * Experience working within Agile delivery methodologies. * Deep expertise in the Databricks Data Intelligence Platform, including Apache Spark, PySpark, Delta Lake, Unity Catalog, Databricks SQL, Lakeflow, Mosaic AI ,AI/BI Genie and Agent Bricks. * Strong experience designing and delivering production-grade batch and streaming data pipelines using modern Databricks engineering patterns. * Advanced knowledge of Spark performance tuning, workload optimisation and Databricks platform cost optimisation. * Strong understanding of Databricks governance, security, enterprise platform architecture, and multi-workspace operating models including Unity Catalog. * Experience implementing CI/CD, automated testing and deployment practices for Databricks, including Infrastructure as Code and Databricks Asset Bundles. * Experience enabling ML and Generative AI workloads using MLflow, Mosaic AI and associated Databricks capabilities. Preferred * Databricks Machine Learning Engineer or Generative AI certifications. Databricks Certified Data Engineer Professional (or equivalent advanced Databricks certification). * Experience with Azure DevOps, GitHub Actions, or equivalent CI/CD tooling. * Experience designing and delivering scalable, production-grade batch and streaming data pipelines on Databricks, including orchestration and integration with enterprise data sources. * Experience optimising and operationalising data pipelines for performance, reliability and cost, including CI/CD, automated testing and monitoring. * Experience with orchestration technologies such as Azure Data Factory, Airflow, or Databricks Workflows. * Experience with Docker, Kubernetes, and containerised deployments. * Consulting experience delivering complex Data & AI engagements across multiple industries. ## Description We're expanding our Databricks Engineering practice and are looking for experienced technical leaders who combine deep engineering expertise, solution architecture capability with strong delivery leadership. You'll work alongside exceptional data, AI, and cloud specialists to design, build, and deliver complex enterprise solutions while leading distributed engineering teams and shaping technical direction across multiple workstreams. Depending on your strengths and client needs, you may focus more on hands-on engineering leadership and delivery, end-to-end solution and platform architecture, or a combination of both across enterprise Databricks transformation programmes. The ideal candidate will possess extensive hands-on expertise across the Databricks Lakehouse Platform, modern data engineering, machine learning, and Generative AI, while providing technical leadership, design authority, and engineering governance throughout the delivery lifecycle. As a key member of our team, you will: * Lead the technical delivery and/or solution architecture of enterprise-scale Databricks Data & AI implementations from solution design through to production deployment. * Act as the technical lead and design authority across multiple workstreams, ensuring solution quality, scalability, security, and alignment with engineering best practices. * Lead and mentor onshore and offshore engineering teams, fostering technical excellence, collaboration, and continuous improvement. * Drive engineering delivery by establishing technical standards, reviewing solution designs, managing technical risks, and ensuring successful execution against project objectives. * Conduct code reviews and engineering quality assurance, promoting clean, maintainable, high-quality code and adherence to software engineering best practices. * Optimise platform performance across Spark workloads, Delta Lake, data pipelines, storage, and compute to maximise scalability, reliability, and cost efficiency. * Design and deliver enterprise-grade Machine Learning, Generative AI and Agentic AI solutions leveraging the Databricks Data Intelligence Platform, MLflow, Mosaic AI, Vector Search, and modern LLM capabilities where appropriate. * Partner with solution architects, product owners, client stakeholders, and engineering teams to translate business requirements into scalable technical solutions. * Coach and develop engineers through technical mentoring, knowledge sharing, and establishing reusable engineering frameworks and best practices. * Own technical governance by maintaining engineering standards, architectural integrity, documentation, and design decisions throughout project delivery. * Define enterprise solution and platform architectures on Databricks - producing reference architectures, blueprints, and target-state designs that span data engineering, ML, and Generative AI. * Support solutioning and pre-sales by shaping proposals, estimating delivery, and advising client leadership on architecture strategy, platform roadmaps, and target operating models., Overall, you're an inspiring technical leader, an experienced engineer, and a delivery-focused problem solver. You understand that successful programmes are built on strong technical leadership, engineering excellence, and collaborative teams. ## Related Videos - [From Messy Queries to Scalable Systems - How Data Engineering actually works](https://www.wearedevelopers.com/videos/100203-from-messy-queries-to-scalable-systems-how-data-engineering-actually-works) - [Innovating Developer Tools with AI: Insights from GitHub Next](https://www.wearedevelopers.com/videos/1268-innovating-developer-tools-with-ai-insights-from-github-next) - [Docker Compose: Rediscovered](https://www.wearedevelopers.com/videos/1978-docker-compose-rediscovered) - [How a Small Team Shrank a Microsoft Monorepo by 94%](https://www.wearedevelopers.com/videos/1236-how-a-small-team-shrank-a-microsoft-monorepo-by-94) - [Modern Data Architectures need Software Engineering](https://www.wearedevelopers.com/videos/1030-modern-data-architectures-need-software-engineering) - [Bringing AI Model Testing and Prompt Management to Your Codebase with GitHub Models](https://www.wearedevelopers.com/videos/1536-bringing-ai-model-testing-and-prompt-management-to-your-codebase-with-github-models) ## Related Articles - [Data Engineer Salary UK](https://www.wearedevelopers.com/magazine/253-data-engineer-salary-uk) - [How to Become an AI Engineer](https://www.wearedevelopers.com/magazine/331-how-to-become-an-ai-engineer) - [Graph and AI Trends 2026: Why Is AI Running but Not Yet Delivering?](https://www.wearedevelopers.com/magazine/680-graph-and-ai-trends-2026-why-is-ai-running-but-not-yet-delivering) - [Coffee with Developers - Maria Apazoglou - Making AI understandable for all in production](https://www.wearedevelopers.com/magazine/475-coffee-with-developers-maria-apazoglou-making-ai-understandable-for-all-in-production) - [Navigating the AI Shift](https://www.wearedevelopers.com/magazine/629-navigating-the-ai-shift) - [Best US AI Conferences for CTOs in 2026: Build vs. Buy, Vendor Evaluation, and Peer Intelligence](https://www.wearedevelopers.com/magazine/736-best-us-ai-conferences-for-ctos-in-2026-build-vs-buy-vendor-evaluation-and-peer-intelligence)