> Markdown version of [/jobs/ext/3061734-senior-data-engineer](https://www.wearedevelopers.com/jobs/ext/3061734-senior-data-engineer). Every page supports `.md` or `Accept: text/markdown`. Links point to the HTML versions so they work for humans too. Agent guide: [/agents.md](https://www.wearedevelopers.com/agents.md). --- # Senior Data Engineer - **Company:** Novatus - **Location:** London, UK - **Experience:** Expert - **Salary:** £68,785.0 - **Contract:** Permanent contract - **Skills:** Clean Code Principles, Airflow, Automation of Tests, Big Data, BigQuery, Cloud Computing, Continuous Integration, Information Engineering, Data Infrastructure, Data Warehousing, DevOps, Python (Programming Language), MySQL, Performance Tuning, Scripting, Freeform SQL, Snowflake, Apache Spark, Electronic Medical Records, Event Driven Architecture, Pyspark, Apache Kafka, Terraform, Data Pipelines, Legacy Systems, Amazon Redshift, Databricks - **Published:** September 25, 2026 - **Apply:** https://www.adzuna.co.uk/jobs/details/5896915373 ## About the Role * 8+ years as a software engineer, including 5+ years in data engineering, owning system design and delivery. * Experience designing auditable, reproducible data pipelines in regulated or high-integrity environments using Python and PySpark. * Able to write and optimize complex SQL queries on large data sets. * Strong data modelling and warehouse design fundamentals. * Strong software engineering fundamentals: clean code, DRY, SOLID, automated testing, CI/CD, and observability, with a bias toward simple, maintainable solutions. * Deep experience with a cloud lakehouse / data warehouse (Databricks, Snowflake, BigQuery, Redshift, EMR, or similar). Our stack is Databricks and Snowflake; strong equivalent experience elsewhere transfers. * Experience with a workflow orchestrator (e.g. Airflow, Dagster, or Databricks Workflows). * Proven system and platform design judgment: you own architectural decisions and can defend the trade-offs. Greenfield, scaling, and migration work all count. * Comfortable translating complex regulatory requirements into technical specifications. * Able to own large, ambiguous initiatives and delivering them in incremental, shippable slices. * Strong communication and collaboration skills, including translating between technical and non-technical stakeholders. Nice-to-Haves: * Experience with event-driven architecture patterns. * Hands-on experience with AWS cloud infrastructure. * Practical experience with infrastructure as code (e.g. Terraform). * Experience modernizing legacy systems. * FinTech, RegTech, or financial services background. ## Description As a Senior Data Engineer, you will be developing our configuration-driven data platform in Databricks, enabling non-engineers to define regulatory logic, and our Snowflake data warehouse, ensuring scalability, auditability, and fitness for client-facing regulatory use cases. You'll be writing clean, maintainable, and well-tested code that follows best practices. As a senior member of the team, you will be providing technical leadership and mentorship to your colleagues. You'll join at a pivotal stage as we modernize our data infrastructure, migrating from Python scripts and MySQL to Databricks and Snowflake. Our systems power regulatory reporting for major financial institutions, requiring precision, traceability, and reliability. Key Responsibilities * Design and evolve our configuration-driven data framework in Databricks and our Snowflake warehouse. * Coach and mentor junior engineers, and lead technical discussions and design reviews. * Design, build, and optimize data pipelines using Databricks, Python and PySpark, with event-driven components (e.g. Kafka). * Ensure pipelines are auditable, lineage-aware, idempotent, and resilient in regulated environments. * Implement robust data quality controls including testing, validation, monitoring, and alerting. * Drive performance optimization across Spark and Snowflake workloads. * Partner with Product, Engineering, DevOps, and Regulatory teams to translate requirements into scalable technical designs. * Contribute to the continuous improvement of our development processes and tools. ## Related Videos - [From Messy Queries to Scalable Systems - How Data Engineering actually works](https://www.wearedevelopers.com/videos/100203-from-messy-queries-to-scalable-systems-how-data-engineering-actually-works) - [From DevOps to Scaled DevOps: How We’re Rebuilding Continuous Delivery as a Platform](https://www.wearedevelopers.com/videos/100018-from-devops-to-scaled-devops-how-we-re-rebuilding-continuous-delivery-as-a-platform) - [MySQL Protocol Features You Should Be Aware Of](https://www.wearedevelopers.com/videos/100267-mysql-protocol-features-you-should-be-aware-of) - [Enjoying SQL data pipelines with dbt](https://www.wearedevelopers.com/videos/823-enjoying-sql-data-pipelines-with-dbt) - [AI Model Management Life Circles: ML Ops For Generative AI Models From Research to Deployment](https://www.wearedevelopers.com/videos/1152-ai-model-management-life-circles-ml-ops-for-generative-ai-models-from-research-to-deployment) - [DevOps Maturity Check – a way to balance autonomy and alignment](https://www.wearedevelopers.com/videos/58-devops-maturity-check-a-way-to-balance-autonomy-and-alignment) ## Related Articles - [Data Engineer Salary UK](https://www.wearedevelopers.com/magazine/253-data-engineer-salary-uk) - [Data Analyst Salary in the UK](https://www.wearedevelopers.com/magazine/278-data-analyst-salary-in-the-uk) - [Software Engineer Salary London](https://www.wearedevelopers.com/magazine/252-software-engineer-salary-london) - [Fullstack Developer Salary UK](https://www.wearedevelopers.com/magazine/251-fullstack-developer-salary-uk) - [Making Data Warehouses Fast: A Developer’s Story](https://www.wearedevelopers.com/magazine/107-making-data-warehouses-fast-a-developer-s-story) - [Best Companies to work for in London: Top 25 Companies in 2023](https://www.wearedevelopers.com/magazine/187-best-companies-to-work-for-in-london-top-25-companies-in-2023)