Senior Data Engineer

• Design, develop, and maintain robust, scalable ETL/ELT pipelines ingesting data from APIs, relational databases, streaming services, and financial data providers
• Implement advanced data processing logic for cleaning, enriching, and aggregating large-scale data using Spark and SQL
• Fine-tune data workloads for maximum performance, throughput, and cloud cost-efficiency
• Drive the implementation of Azure Databricks services, leveraging Unity Catalog and Delta Lake architectures
• Develop and maintain data solutions using Python, SQL, R, YAML, and JavaScript
• Lead hands-on implementation of Azure Purview to manage data quality, governance, metadata, and end-to-end data lineage tracking
• Establish automated data validation, quality checks, and real-time alerting processes across all production environments
• Partner with DevOps teams to design, build, and maintain robust CI/CD pipelines for automated environmental deployments
• Collaborate closely with economists, data scientists, and senior analysts to translate complex analytical needs into production-ready data systems
• Drive engineering excellence through active participation in code reviews, architectural discussions, and knowledge-sharing sessions

Technologies:

• Azure
• CI/CD
• Cloud
• Databricks
• DevOps
• Docker
• ETL
• Git
• JavaScript
• Kafka
• Kubernetes
• NoSQL
• Python
• PySpark
• SQL
• Scala
• Spark
• Unity
• GameDev
• Security

• 10+ years of dedicated data engineering experience managing massive, complex datasets
• 3+ years of deep, hands-on production experience with Azure Databricks, Spark (PySpark/Scala), and Delta Lake ecosystems
• Extensive experience across the Azure data suite, specifically Azure Data Factory, Azure Blob Storage, and Azure SQL Database
• Strong proficiency in Python, Spark, and SQL
• Practical experience using Azure Purview for governance and cataloguing
• Working knowledge or exposure to R, YAML, and JavaScript within data workflows
• Experience with event-driven data such as Kafka or Azure Event Hubs and modern DevOps tooling including Azure DevOps, Git, Docker, and Kubernetes
• Solid understanding of both SQL and NoSQL database design, data warehousing principles, and data modelling techniques
• Experience working within financial services, central banking, or an economic data environment is highly advantageous
• Exceptional ability to bridge the gap between technical infrastructure and economic or analytical business logic
• Microsoft Certified: Azure Data Engineer Associate or Databricks certifications are preferred
• SC Clearance required, active or eligible to undergo

We are a London-based team working in a hybrid setup of 3 days in the office and 2 days remote. We are building and scaling our Azure Databricks platform to power our Monetary Analysis, Forecasting, and Modelling frameworks, with a focus on secure, reliable, and high-performance data infrastructure. This is a full-time, permanent senior role with a salary of £75,000 to £80,000 plus benefits and bonus, and it offers the opportunity to work closely with economists, data scientists, and senior analysts in a business-critical environment.