Data Ingestion Engineer

We are looking for an experienced Senior Data Engineer to join a high-performing data engineering team responsible for large-scale data ingestion and processing.

You'll play a key role in ensuring critical data pipelines remain reliable, scalable and efficient, working with high-volume datasets that support analytics, data science and machine learning initiatives.

This is a hands-on engineering role focused on solving complex production challenges, improving data pipeline resilience and optimising distributed data processing systems., * Build, maintain and optimise large-scale data ingestion and ETL pipelines using Apache Spark and Python.

• Debug and resolve failing or blocked data ingestion pipelines in production.
• Investigate issues caused by corrupt, malformed or unexpected data and implement robust solutions to prevent pipeline failures.
• Improve pipeline resilience so individual data quality issues do not interrupt wider workflows.
• Process and transform large datasets from multiple internal and third-party data sources.
• Design and improve handling of varied data formats from partners, suppliers and external providers.
• Support orchestration across multi-stage ingestion workflows, including dependency management, retries and queue handling.
• Optimise Spark jobs for throughput, reliability and compute efficiency.
• Reduce operational overhead by minimising failed jobs, stalled pipelines and manual intervention.
• Work closely with engineering teams to ensure critical datasets are delivered to downstream analytics, data science and machine learning teams.
• Contribute practical improvements that enhance the scalability, reliability and maintainability of the data platform.

• Strong commercial experience with Apache Spark (PySpark) in production environments.
• Excellent Python development skills.
• Proven experience designing, building and supporting large-scale data ingestion, ETL or data processing pipelines.
• Experience working with distributed data processing technologies.
• Strong understanding of data ingestion, transformation and loading processes.
• Experience diagnosing and resolving production pipeline failures.
• Comfortable working with incomplete, inconsistent or corrupt datasets.
• Experience optimising data processing jobs for performance, scalability and cost efficiency.
• Understanding of workflow orchestration, multi-stage pipelines and downstream dependencies.
• Ability to work independently in a fast-paced, highly technical environment.
• A pragmatic, delivery-focused approach with excellent problem-solving skills.
• Experience working with very large datasets, ideally in petabyte-scale or other high-throughput environments.

Desirable Experience:

• Experience with workflow orchestration tools such as Airflow, Dagster or similar.
• Cloud platform experience (AWS, Azure or GCP).
• Knowledge of modern data lake or data warehouse architectures.
• Experience supporting machine learning or data science platforms.