Data Engineer

AD Data Analytics

Peterborough, UK

2 months ago

Role details

Contract type

Permanent contract

Employment type

Full-time (> 32 hours)

Experience level

Expert

Experience required

5 years minimum

Compensation

£50,000.0 - £55,000.0

Working hours

Regular working hours

Job source

indeed.com

Tech stack

Java (Programming Language) Computer Programming Information Engineering Data Infrastructure Extract Transform Load (ETL) Data Mining Relational Databases Python (Programming Language) SQL Databases Data Logging Data Processing Azure Data Factory

+11 more

System Availability Apache Spark Git Data Analytics Apache Kafka Spark Streaming Data Management Stream Processing Software Version Control Data Pipelines Databricks

Job description

We are seeking a skilled and motivated Data Engineer to join our dynamic team. In this role, you will be responsible for designing, building, and maintaining robust data pipelines and architectures that support our data-driven decision-making processes. You will work closely with data scientists, analysts, and other stakeholders to ensure that our data infrastructure is efficient, scalable, and aligned with business needs., * Design and implement robust data pipelines using Azure Data Factory (ADF) and Databricks.

Develop scalable ETL/ELT processes to ingest, transform, and load data from various sources.
Write clean, efficient, and reusable Python code for data processing and automation.
Optimize and manage SQL queries for data extraction, transformation, and reporting.
Collaborate with business stakeholders to understand data requirements and deliver solutions aligned with business goals.
Ensure data quality, integrity, and security across all data platforms.
Monitor and troubleshoot data pipelines to ensure high availability and performance.
Implement logging, alerting, and error-handling mechanisms for proactive issue resolution
Apply domain knowledge of trading and financial markets to enhance data models and analytics capabilities.

Requirements

Do you have experience in Spark?, * Proficiency in programming languages such as Java, Python, and SQL.

5+ years of experience in data engineering or related roles.
Proven expertise in Azure Data Factory and Azure Databricks.
Strong proficiency in Python for data engineering tasks.
Solid understanding of SQL and relational databases.
Exposure to real-time data processing frameworks (e.g., Spark Streaming, Kafka).
Experience in maintaining and supporting production-grade data pipelines.
Experience with version control systems (e.g., Git) and CI/CD pipelines.
Excellent problem-solving skills and attention to detail.
Strong communication and collaboration abilities
Excellent problem-solving skills coupled with strong communication abilities. If you are passionate about leveraging technology to drive business insights through effective data management, we encourage you to apply for this exciting opportunity as a Data Engineer.