Lead Data Engineer

Inara
Manchester, United Kingdom
2 days ago
Apply on www.apply4u.co.uk
Prepare application

Role details

Contract type
Permanent contract
Employment type
Full-time (> 32 hours)
Experience level
Expert
Compensation
£80,000 - £85,000
Working hours
Regular working hours

Tech stack

Sql Server Data Tools (SSDT) Amazon S3 Batch Processing Continuous Integration Customer Data Management Data Architecture Data Governance Identity and Access Management Python (Programming Language) Metadata Microsoft SQL Server PCI Data Security Standards
+13 more
Power BI Standard Sql SQL Server Integration Services Great Expectations (Foster Youth College-readiness and Support Program) Large Language Models Claude Code Amazon Relational Database Service Pyspark Apache Flink AWS Data Analytics Claude Cloudwatch Terraform

Job description

What You’ll NeedProven delivery in both batch processing (warehouse loads, ELT, orchestration) and modern streaming, with Flink or comparable engines

Requirements

Production AWS data engineering experience: S3, Lambda, Glue, Athena/Iceberg, Kinesis or equivalent streaming services, RDS, IAM, CloudWatch, plus infrastructure as code with Terraform and CI/CD Strong lakehouse and data modelling depth: Iceberg-style open table formats, medallion-style layering, dimensional modelling, and the judgement to know when a warehouse should consume from a lakehouse rather than replicate it Strong SQL and Python with proper testing and engineering discipline. Willingness to work with SQL Server, SSIS/SSDT and Power BI at the start, even if you’ve moved beyond them elsewhere Experience designing semantic layers, conformed metric definitions and glossaries, and exposing data to LLM tooling through MCP servers or similar Daily hands-on use of Claude Code or an equivalent frontier coding assistant as part of how you engineer, with a clear story of how it’s made you materially faster without compromising quality or security Solid data governance experience in a regulated business: retention, lineage, classification, PII minimisation, auditability and PCI-DSS boundaries built into the platform, not bolted on Business-area fluency, learning what functions like FinCrime, Regulatory Reporting, Product, CX, Finance and Marketing actually do with data, and prioritising from their outcomes Experience line-managing a small technical team in a delivery setting, including hiring and development Clear written and spoken communication with senior stakeholders, including architecture position papers and candid risk statements

Bonus Points ForExperience in financial services or another regulated environment PySpark for large-scale batch or streaming transformations Migrating a SQL Server/SSIS estate to a metadata-driven or lakehouse architecture, and a clear view on what you’d do differently Designing or operating a Customer Data Platform or event platform (Kinesis, Flink, StarRocks or comparable) Building data-quality and data-contract tooling (dbt tests, Great Expectations or equivalent) and synthetic monitoring around feeds Authoring Claude skills or MCP servers for a team, or running an LLM natural-language layer over BI data FinCrime, fraud or regulatory-reporting data domains Power BI at the platform level (semantic models, gateway, administration) for the transition period

About the company

Salary: £80,000 - £85,000 (potentially some wiggle room for the right person) We’re working with an established fintech business in Greater Manchester that is doing something genuinely interesting with AI right now. Not pilots. Not proof of concepts gathering dust. Real, production-grade AI embedded into customer journeys and internal operations, and they’re only just getting started. They’ve recently brought in a Head of Data and AI who has both the vision and the organisational backing to push this forward hard. This role sits right at the centre of that push, owning the data platform that everything else gets built on. The RoleThis is a hands-on Lead Data Engineer role reporting into the Head of Data & AI. You’ll take on the existing data engineering estate and bring state-of-the-art batch and live data processing to it, so it can properly serve BI, Data Science and Agentic AI use cases going forward. It’s a lead role with line management, but a genuinely hands-on one. You’ll be expected to use tools like Claude Code as a core part of how you engineer and build, push the technology envelope where it’s warranted, and bring a small technical team with you as you do it. The aim isn’t to rebuild for the sake of it. It’s to evolve what works, automate what shouldn’t be manual, and drive towards one platform that serves the whole business rather than a fragmented, function-based estate. What You’ll Be DoingImproving and evolving the existing data estate so it’s ready for modern batch and live-streaming data processing Simplifying and automating data processes that are manual, duplicated or fragile today Supporting the BI, Data Science and Agentic (Applied AI) teams from a single platform Building a semantic layer over the data and exposing it through MCP and similar interfaces, giving the business the ability to analyse its data through tools like Claude Delivering a governed self-serve capability that lets anyone in the business query data safely Leading the design of a Customer Data Platform for batch and live data processing, working closely with ML and Data Science on shared ingestion, storage and feature needs Line-managing the data engineers, including hiring and developing the team needed to deliver Owning the governance, security and running cost of the data platform alongside Data & AI leadership

Apply for this position

This job is hosted externally. Click below to view the full posting and apply.

Apply on www.apply4u.co.uk
Prepare application

Good distractions

Loading talks and stories from around this role…