Senior Data Engineer

• Design, develop, and maintain robust, scalable ETL/ELT pipelines ingesting data from APIs, relational databases, streaming services, and financial data providers.
• Implement advanced data processing logic for cleaning, enriching, and aggregating large-scale data using Spark and SQL.
• Fine-tune data workloads for maximum performance, throughput, and cloud cost-efficiency.
• Drive the implementation of Azure Databricks services, leveraging Unity Catalog and Delta Lake architectures.
• Develop and maintain data solutions using Python, SQL, R, YAML, and JavaScript.
• Lead hands-on implementation of Azure Purview to manage data quality, governance, metadata, and end-to-end data lineage tracking.
• Establish automated data validation, quality checks, and real-time alerting processes across production environments.
• Partner with DevOps teams to design, build, and maintain robust CI/CD pipelines for automated environmental deployments.
• Collaborate closely with economists, data scientists, and senior analysts to translate complex analytical needs into production-ready data systems.
• Drive engineering excellence through code reviews, architectural discussions, and knowledge-sharing sessions.

Technologies:

• Azure
• CI/CD
• Cloud
• Databricks
• DevOps
• Docker
• ETL
• Git
• JavaScript
• Kafka
• Kubernetes
• NoSQL
• Python
• PySpark
• SQL
• Scala
• Spark
• Unity
• GameDev
• Security

More:

We are a London-based team working in a hybrid setup with 3 days in the office and 2 days remote. We are offering a full-time, permanent Senior Data Engineer role focused on our Monetary Analysis, Forecasting, and Modelling frameworks, with a salary of £75,000 to £80,000 plus benefits and bonus. Our cutting-edge Azure Databricks platform is central to our mission, powering secure, reliable, business-critical economic data systems in a collaborative environment where we work closely with economists, data scientists, and senior analysts.

• 10+ years of dedicated data engineering experience managing large, complex datasets.
• 3+ years of hands-on production experience with Azure Databricks, Spark (PySpark/Scala), and Delta Lake.
• Extensive experience across the Azure data suite, including Azure Data Factory, Azure Blob Storage, and Azure SQL Database.
• Strong proficiency in Python, Spark, and SQL.
• Practical experience using Azure Purview for governance and cataloguing.
• Working knowledge or exposure to R, YAML, and JavaScript within data workflows.
• Experience with event-driven data platforms such as Kafka or Azure Event Hubs.
• Experience with modern DevOps tooling such as Azure DevOps, Git, Docker, and Kubernetes.
• Solid understanding of SQL and NoSQL database design, data warehousing principles, and data modelling techniques.
• Experience working within financial services, central banking, or an economic data environment is highly advantageous.
• Exceptional ability to bridge the gap between technical infrastructure and economic or analytical business logic.
• Microsoft Certified: Azure Data Engineer Associate or Databricks certifications are preferred.
• Active SC Clearance or eligibility to undergo SC Clearance.