Data Engineer

AD Data Analytics
Peterborough, United Kingdom
2 days ago

Role details

Contract type
Permanent contract
Employment type
Full-time (> 32 hours)
Working hours
Regular working hours
Languages
English
Experience level
Senior
Compensation
£ 55K

Job location

Peterborough, United Kingdom

Tech stack

Java
Computer Programming
Information Engineering
Data Infrastructure
ETL
Data Mining
Relational Databases
Python
SQL Databases
Data Logging
Data Processing
Azure
System Availability
Spark
GIT
Data Analytics
Kafka
Spark Streaming
Data Management
Stream Processing
Software Version Control
Data Pipelines
Databricks

Job description

We are seeking a skilled and motivated Data Engineer to join our dynamic team. In this role, you will be responsible for designing, building, and maintaining robust data pipelines and architectures that support our data-driven decision-making processes. You will work closely with data scientists, analysts, and other stakeholders to ensure that our data infrastructure is efficient, scalable, and aligned with business needs., * Design and implement robust data pipelines using Azure Data Factory (ADF) and Databricks.

  • Develop scalable ETL/ELT processes to ingest, transform, and load data from various sources.
  • Write clean, efficient, and reusable Python code for data processing and automation.
  • Optimize and manage SQL queries for data extraction, transformation, and reporting.
  • Collaborate with business stakeholders to understand data requirements and deliver solutions aligned with business goals.
  • Ensure data quality, integrity, and security across all data platforms.
  • Monitor and troubleshoot data pipelines to ensure high availability and performance.
  • Implement logging, alerting, and error-handling mechanisms for proactive issue resolution
  • Apply domain knowledge of trading and financial markets to enhance data models and analytics capabilities.

Requirements

Do you have experience in Spark?, * Proficiency in programming languages such as Java, Python, and SQL.

  • 5+ years of experience in data engineering or related roles.
  • Proven expertise in Azure Data Factory and Azure Databricks.
  • Strong proficiency in Python for data engineering tasks.
  • Solid understanding of SQL and relational databases.
  • Exposure to real-time data processing frameworks (e.g., Spark Streaming, Kafka).
  • Experience in maintaining and supporting production-grade data pipelines.
  • Experience with version control systems (e.g., Git) and CI/CD pipelines.
  • Excellent problem-solving skills and attention to detail.
  • Strong communication and collaboration abilities
  • Excellent problem-solving skills coupled with strong communication abilities. If you are passionate about leveraging technology to drive business insights through effective data management, we encourage you to apply for this exciting opportunity as a Data Engineer.

Benefits & conditions

Pulled from the full job description

  • Flexitime
  • Employee mentoring programme
  • Health & wellbeing programme

Apply for this position