> Markdown version of [/jobs/ext/2803737-lead-data-architect](https://www.wearedevelopers.com/jobs/ext/2803737-lead-data-architect). Every page supports `.md` or `Accept: text/markdown`. Links point to the HTML versions so they work for humans too. Agent guide: [/agents.md](https://www.wearedevelopers.com/agents.md). --- # Lead Data Architect - **Company:** Collage Recruitment - **Location:** Greater London, UK - **Experience:** Expert - **Contract:** Permanent contract - **Skills:** Artificial Intelligence, Automation of Tests, Spreadsheets, Continuous Integration, Data Architecture, Python (Programming Language), Microsoft SQL Server, Standard Sql, SQL Stored Procedures, SQL Databases, Cloud Platform System, Apache Spark, Pyspark, Infrastructure Automation Frameworks, Databricks - **Published:** September 9, 2026 - **Apply:** https://www.collegerecruiter.com/job/2842869067-lead-data-architect ## About the Role * Hands-on experience delivering a Databricks migration at production scale - this is the core of the role * Strong stakeholder and relationship management skills. You'll work closely with business leadership, analysts, and engineers across regions, often with ambiguity to navigate * Advanced Python, PySpark, and SQL * A track record migrating SQL Server workloads, stored procedures, and scheduled jobs to modern, cloud-native platforms * Solid understanding of Unity Catalog governance, security models, and access control * Experience with CI/CD, automated testing, and Infrastructure as Code, * Experience in a smaller, B2B-focused fintech environment - this role suits someone used to close commercial context and hands-on delivery, rather than large enterprise-scale environments with extensive legacy system sprawl * Insurance, financial services, pricing, or marketing analytics background * Familiarity with UK GDPR, data residency, and regional governance * dbt or analytics engineering experience ## Description Our client, a leading provider of benchmark data and analytics for the financial services sector, is hiring the first senior engineering leader for their UK Insurance data business. This is a newly created role, not a backfill. This is a rare opportunity to own a platform transformation from the ground up: migrating a legacy SQL benchmarking engine onto a modern Databricks Lakehouse, and then building the product tooling on top of it. Our client provides benchmark data behind pricing, acquisition, renewal, and retention decisions across the UK insurance market, built on a proprietary, behavioural dataset that underpins trusted relationships with senior commercial decision-makers across the industry. You'll report directly to the Head of Insurance, based in London, and will be the senior technical voice for the UK Insurance business - with the option to line-manage an existing mid-level engineer as the team grows. You'll also work closely with a broader, global engineering function on platform standards, while owning delivery locally., * Lead the migration of legacy SQL solutions into Databricks Jobs, Workflows, and PySpark/SQL pipelines, starting with a lift-and-shift of core benchmark data, while maintaining operational SLAs throughout * Define the target-state data architecture (Lakehouse, Medallion, domain-oriented data products) to support reporting, analytics, and future AI/ML use cases * Work directly with business leadership and analysts to codify canonical metric logic - replacing spreadsheet calculations with documented, testable code * Own UK Unity Catalog governance: catalogue, schema, table design, ownership, and access control * Build automated QA and reconciliation between legacy and new outputs, with the observability to give internal and external stakeholders confidence in the data * Implement CI/CD and Infrastructure as Code for Databricks assets * Tune Spark and SQL workloads for performance and cost * Act as the primary link between UK delivery and the centralised platform engineering function * Provide technical leadership and mentorship, with the option to line-manage a mid-level engineer ## Related Videos - [PySpark - Combining Machine Learning & Big Data](https://www.wearedevelopers.com/videos/44-pyspark-combining-machine-learning-big-data) - [Enjoying SQL data pipelines with dbt](https://www.wearedevelopers.com/videos/823-enjoying-sql-data-pipelines-with-dbt) - [Launching a marketplace on-time: A lesson in taking shortcuts using spreadsheets!](https://www.wearedevelopers.com/videos/477-launching-a-marketplace-on-time-a-lesson-in-taking-shortcuts-using-spreadsheets) - [Cutting LLM Costs Without Cutting Quality: How to Beat Proprietary LLMs with Fine-Tuned Open Source](https://www.wearedevelopers.com/videos/100151-cutting-llm-costs-without-cutting-quality-how-to-beat-proprietary-llms-with-fine-tuned-open-source) - [OLTP in the Lakehouse: Redefining Data for AI Workloads](https://www.wearedevelopers.com/videos/2038-oltp-in-the-lakehouse-redefining-data-for-ai-workloads) - [Empowering Retail Through Applied Machine Learning](https://www.wearedevelopers.com/videos/976-empowering-retail-through-applied-machine-learning) ## Related Articles - [Data Engineer Salary UK](https://www.wearedevelopers.com/magazine/253-data-engineer-salary-uk) - [Data Analyst Salary in the UK](https://www.wearedevelopers.com/magazine/278-data-analyst-salary-in-the-uk) - [How to Become an AI Engineer](https://www.wearedevelopers.com/magazine/331-how-to-become-an-ai-engineer) - [Navigating the AI Shift](https://www.wearedevelopers.com/magazine/629-navigating-the-ai-shift) - [Graph and AI Trends 2026: Why Is AI Running but Not Yet Delivering?](https://www.wearedevelopers.com/magazine/680-graph-and-ai-trends-2026-why-is-ai-running-but-not-yet-delivering) - [Making Data Warehouses Fast: A Developer’s Story](https://www.wearedevelopers.com/magazine/107-making-data-warehouses-fast-a-developer-s-story)