Big Data Engineer Python

This role focuses on developing, integrating, and optimising large-scale data flows. It involves close collaboration with data scientists to translate analytical models into practical technical solutions, requiring hands-on expertise in data cleansing, pipeline design, component orchestration, and modern data technologies such as Hadoop, Spark, and API development.

Mission & Responsibilities

• Technical Collaboration: work closely with data scientists to develop effective technical solutions

• Data Optimisation: perform data cleansing, preparation, and integration

• Pipeline Engineering: design and build robust data pipelines

• System Integration: manage component orchestration

• Large-Scale Data Processing: responsible for batch Spark ETL, including data wrangling, joining, and aggregation

Technical Competencies

• AI Ecosystems & Frameworks, including:

• Modern Data Stack (MDS)

• On-premises Data Lake technologies (Hadoop, Spark, Hive)

• Large-scale data processing (Spark)

• Orchestration tools (Apache Airflow)

• Containerisation (Docker / Kubernetes)

• Python programming

• Architecture & Integrations:

• API development

• Cloud architectures (AWS, Azure)

Experience with real-time architectures and event streaming infrastructure is advantageous.

Behavioural Competencies

• Collaboration: able to work closely and effectively with data scientists to translate complex analytical needs into functional technical solutions

• Analytical & Organised: precise and methodical approach to data cleansing, preparation, integration, and pipeline design

• Problem Solving: quick and efficient in addressing bottlenecks in data processing or orchestration errors

We require full-time availability (100%), although 80% availability will be considered.