Lead Data Engineer

We are looking for a Senior / Lead Data Engineer to join the engineering organisation a multi-year banking transformation programme delivering capabilities across Accounts (Real-Time Ledger), Payments, Foreign Exchange and Client Lifecycle Management, underpinned by an Enterprise Data Platform built on Snowflake.

This is a hands-on build role. Working closely with the Enterprise Data Architect, you will turn the target-state data architecture into a production platform: standing up the platform foundations, engineering data acquisition and transformation, and implementing canonical data models and data products. You will act as the engineering counterpart to the Data Architect implementing the design, feeding back on buildability, and setting the engineering standards others follow. At Lead level, you will also shape and lead the data engineering team.

Key Responsibilities

Platform foundations

Build Snowflake environment structures layered databases and schemas (Raw / Conformed / Consumption), environment separation (development / test / production) and naming conventions

Implement role-based access control, service accounts, virtual warehouse design, resource monitors and cost guardrails

Manage the platform as code Snowflake objects, roles, grants and warehouses through infrastructure-as-code and version control

Establish Git workflows and CI/CD pipelines for data code, with automated build, test and deployment across environments

Data acquisition and CDC

Assess source systems and recommend fit-for-purpose acquisition patterns full vs incremental, batch vs micro-batch vs event, and file vs API vs direct database vs message hub

Advise on where change data capture should be applied and owned log-based CDC at source, application-emitted change events, or warehouse-side change detection and implement the chosen patterns

Engineer CDC and replication from relational, document and time-series application databases, handling initial loads, deletes, schema evolution, lag and backfills

Ingest event streams from messaging platforms into Snowflake, covering ordering, idempotency, deduplication, replay and error handling

Configure and optimise managed ELT connectors (e.g. Fivetran), and build file- and API-based ingestion where required

Design and build a metadata-driven ingestion framework sources, objects, load types (full, incremental, CDC), schedules and target mappings held as configuration, with control tables for watermarks, run status, audit and reprocessing so new feeds are onboarded through metadata rather than new code

Make the framework reusable across ingestion channels (managed ELT, file, API, database CDC and event feeds), with generic, parameterised loading, schema-drift handling and standard error handling

Prove patterns through technical spikes, and record decisions as Architecture Decision Records

Transformation and modelling

Design and build dbt projects from the ground up structure, layering, conventions, macros, packages, tests and documentation

Run dbt both natively in Snowflake (dbt Projects on Snowflake) and on dbt Cloud, building portable, convention-led projects that move between the two with minimal rework

Implement canonical data models including industry-aligned models such as BIAN-based business objects from source-to-canonical mapping specifications and governed reference data

Build dimensional and domain-oriented data products for reporting, regulatory, analytics and AI consumers

Engineer identity resolution, key management and cross-source reconciliation, and feed implementation findings back into the models and mappings

Quality, governance and operations

Embed data quality as code tests, freshness checks, reconciliations and data contract enforcement

Implement Snowflake governance controls tagging and classification, dynamic data masking, row access policies and PII handling aligned to GDPR with lineage that supports BCBS 239

Build monitoring and alerting for pipeline failures, freshness, data quality and cost, and define run-books and support processes

Orchestration and scheduling

Design and implement orchestration and scheduling natively in Snowflake using Tasks and task graphs (DAGs) dependencies, root and finaliser tasks, serverless vs warehouse-backed compute, schedules and event-triggered runs

Orchestrate ingestion, dbt execution (e.g. EXECUTE DBT PROJECT), data quality checks and downstream refreshes end to end within Snowflake, with retries, failure handling, alerting and notifications

Use Dynamic Tables where declarative, target-lag-driven pipelines are a better fit than task-based scheduling, choosing between incremental and full refresh modes and managing their cost

Build operational visibility over task and Dynamic Table runs run history, latency, failures and credit consumption and keep orchestration designs portable to external orchestrators where needed

AI-assisted engineering

Use Snowflake CoCo (formerly Cortex Code) to accelerate development, with generated code reviewed, tested and deployed through standard CI/CD

Apply Snowflake Cortex AI capabilities AISQL functions, Cortex Search, semantic views, Cortex Analyst and Cortex Agents where they add value, and design data products to be AI-ready

Leadership and collaboration

Work as the engineering counterpart to the Enterprise Data Architect, contributing to design decisions and ADRs

Collaborate with Integration, Solution and Application teams on source extracts, event contracts and data contracts

Define and own engineering standards, templates and reusable patterns

Lead level: lead, mentor and help recruit data engineers, own code review, and plan and report delivery

Senior: 10+ years in data engineering, including at least 3 years of hands-on Snowflake delivery in production. Lead: 12+ years, including leading a data engineering team or workstream

Strong track record in data warehousing and data modelling, with experience implementing models designed by architects as well as designing their own

Has stood up a new data platform or major platform workstream environments, access control, automation rather than only extending an existing one

Has built a dbt project from the ground up, including standards, testing and CI/CD

Has designed and built a metadata-driven ingestion framework that has been used to onboard multiple sources in production

Strong hands-on experience scheduling and orchestrating production pipelines natively in Snowflake with Tasks / task graphs and Dynamic Tables

Has designed and implemented data acquisition across heterogeneous sources, and can clearly explain the trade-offs between full, incremental and CDC approaches

Comfortable reading and implementing architecture artefacts canonical models, mapping specifications, ADRs and offering well-reasoned implementation feedback

Clear communicator, able to work with architects, application teams and business stakeholders