> Markdown version of [/jobs/ext/324314-hadoop-spark-data-engineer](https://www.wearedevelopers.com/jobs/ext/324314-hadoop-spark-data-engineer). Every page supports `.md` or `Accept: text/markdown`. Links point to the HTML versions so they work for humans too. Agent guide: [/agents.md](https://www.wearedevelopers.com/agents.md).

---

# Hadoop Spark Data Engineer

- **Company:** Capgemini
- **Location:** London, UK (Remote available)
- **Contract:** Permanent contract
- **Skills:** Amazon Web Services, Microsoft Azure, Big Data, Computer Programming, Data Integration, Extract Transform Load (ETL), Data Security, Distributed Data Store, Distributed Systems, Apache Hadoop, Hadoop Distributed File System, MapReduce, Apache HBase, Apache Hive, Performance Tuning, Standard Sql, SQL Databases, Sqoop, Data Streaming, Apache Yarn, Apache Spark, Apache Flume, Apache Kafka, Functional Programming, Stream Processing
- **Published:** June 1, 2026
- **Apply:** https://uk.indeed.com/viewjob?jk=68d141a0555094e0

## About the Role

Do you have experience in Spark?, Results-driven Hadoop Spark Data Engineer with strong expertise in designing and implementing scalable big data solutions using Scala and Apache Spark. Experienced in working with the Hadoop ecosystem, including HDFS, Hive, and YARN, to process and analyze large datasets efficiently. Skilled in building robust ETL pipelines, real-time data processing, and optimizing distributed systems for performance and reliability. Proficient in SQL, data modeling, and integrating data from multiple sources in cloud and on-prem environments., * Proficient in Scala programming with strong expertise in functional programming concepts for building scalable data applications.

 * Extensive experience in Apache Spark (Core, SQL, and Streaming) for processing large-scale distributed data efficiently
 * Strong knowledge of Hadoop ecosystem components including HDFS, YARN, Hive, and HBase.
 * Skilled in designing and developing ETL pipelines and handling structured and unstructured big data.
 * Experienced in performance tuning, data optimization, and working with distributed systems in cloud or on-prem environments.

We are a Disability Confident Employer:

Capgemini is proud to be a Disability Confident Employer (Level 2) under the UK Government's Disability Confident scheme. As part of our commitment to inclusive recruitment, we will offer an interview to all candidates who:

 * Declare they have a disability, and
 * Meet the minimum essential criteria for the role.

## Description

* Build operate monitor and troubleshoot Hadoop clusters.
 * Write scalable ETL processes using tools like Hive Pig and Spark.
 * Develop and maintain data ingestion processes using Sqoop Flume or Kafka.
 * Optimize MapReduce jobs and manage HDFS storage.
 * Collaborate with data scientists and analysts to support data needs.
 * Ensure data security and compliance with organizational policies.
 * Create and maintain technical documentation and playbooks.
 * Evaluate and integrate cloudbased big data solutions AWS GCP Azure.


## Related Videos

- [PySpark - Combining Machine Learning & Big Data](https://www.wearedevelopers.com/videos/44-pyspark-combining-machine-learning-big-data)
- [Alibaba Big Data and Machine Learning Technology](https://www.wearedevelopers.com/videos/37-alibaba-big-data-and-machine-learning-technology)
- [Serverless: Past, Present and Future](https://www.wearedevelopers.com/videos/34-serverless-past-present-and-future)
- [Why and when should we consider Stream Processing frameworks in our solutions](https://www.wearedevelopers.com/videos/1085-why-and-when-should-we-consider-stream-processing-frameworks-in-our-solutions)
- [From Messy Queries to Scalable Systems - How Data Engineering actually works](https://www.wearedevelopers.com/videos/100203-from-messy-queries-to-scalable-systems-how-data-engineering-actually-works)
- [Let's Get Aggregated: Custom UDAFs in Spark  ](https://www.wearedevelopers.com/videos/1649-let-s-get-aggregated-custom-udafs-in-spark)

## Related Articles

- [Data Engineer Salary UK](https://www.wearedevelopers.com/magazine/253-data-engineer-salary-uk)
- [Top Big Data Technologies That You Need to Know](https://www.wearedevelopers.com/magazine/108-top-big-data-technologies-that-you-need-to-know)
- [Data Analyst Salary in the UK](https://www.wearedevelopers.com/magazine/278-data-analyst-salary-in-the-uk)
- [Software Engineer Salary London](https://www.wearedevelopers.com/magazine/252-software-engineer-salary-london)
- [Find a Developer Job: 12 Best Job Sites For Developers](https://www.wearedevelopers.com/magazine/165-find-a-developer-job-12-best-job-sites-for-developers)
- [Best Companies to work for in London: Top 25 Companies in 2023](https://www.wearedevelopers.com/magazine/187-best-companies-to-work-for-in-london-top-25-companies-in-2023)