> Markdown version of [/jobs/ext/3810652-senior-data-engineer](https://www.wearedevelopers.com/jobs/ext/3810652-senior-data-engineer). Every page supports `.md` or `Accept: text/markdown`. Links point to the HTML versions so they work for humans too. Agent guide: [/agents.md](https://www.wearedevelopers.com/agents.md).

---

# Senior Data Engineer

- **Company:** GCS Ltd
- **Location:** London, UK
- **Experience:** Expert
- **Salary:** £87,253.0
- **Contract:** Temporary contract
- **Skills:** Agile Methodology, Airflow, Amazon Web Services, Microsoft Azure, Big Data, Cloud Computing, Software Quality, Code Reuse, Code Review, Continuous Integration, Data Integration, Extract Transform Load (ETL), Data Systems, DevOps, Distributed Computing Environment, Distributed Systems, Python (Programming Language), Machine Learning, Scrum Methodology, Standard Sql, Software Engineering, Workflow Management Systems, Enterprise Data Management, Data Processing, Apache Spark, Data Lakes, Pyspark, Kubernetes, Deployment Automation, Integration Frameworks, Apache Kafka, Video Streaming, Stream Processing, Software Version Control, Data Pipelines, Docker, Databricks
- **Published:** October 18, 2026
- **Apply:** https://www.adzuna.co.uk/jobs/details/5928850303

## About the Role

* Strong commercial experience as a Data Engineer or Senior Data Engineer
 * Expert-level Python development skills
 * Extensive hands-on experience with Databricks in enterprise environments
 * Strong Apache Spark and PySpark expertise
 * Advanced SQL development and optimisation skills
 * Experience designing and building large-scale ETL and data integration solutions
 * Strong understanding of distributed computing concepts and data processing frameworks
 * Experience with modern cloud platforms, including Azure, AWS or GCP
 * Knowledge of software engineering principles, testing and code quality practices
 * Experience working in Agile development environments
 * Excellent problem-solving and troubleshooting capabilities
 * Desirable: experience in investment banking, financial services or other highly regulated environments
 * Desirable: exposure to risk, trading, treasury or regulatory data domains
 * Desirable: experience with Kafka, streaming technologies or real-time data processing
 * Desirable: Databricks certifications
 * Desirable: experience with Airflow, Databricks Workflows or similar orchestration tools
 * Desirable: CI/CD pipeline development and DevOps practices
 * Desirable: experience with Docker and Kubernetes
 * Desirable: experience supporting data science or machine learning workloads

## Description

* Design, develop and maintain scalable data pipelines using Python, Databricks and PySpark
 * Build and optimise ETL/ELT solutions that process large-scale datasets
 * Develop high-quality, reusable code and engineering frameworks
 * Deliver robust, performant and maintainable data solutions
 * Troubleshoot and resolve complex production and performance issues
 * Enhance and modernise enterprise data platforms
 * Develop and support Databricks-based data solutions and Lakehouse architectures
 * Use Delta Lake and Spark to deliver scalable data processing capabilities
 * Optimise distributed data processing workloads for performance and reliability
 * Implement data quality controls and monitoring solutions
 * Contribute to platform stability, scalability and operational excellence
 * Work with business and technology stakeholders to understand requirements and deliver effective solutions
 * Participate in Agile delivery activities, including sprint planning, refinement and estimation
 * Support strategic data initiatives across multiple business areas
 * Collaborate with architects, engineers and product teams to deliver high-quality outcomes
 * Identify and resolve technical risks and implementation challenges
 * Promote engineering best practices, including testing, code reviews and version control
 * Support CI/CD adoption and deployment automation
 * Contribute to coding standards and engineering governance
 * Improve development and support processes
 * Mentor junior engineers and share knowledge within the team

Technologies:

 * Airflow
 * AWS
 * Azure
 * CI/CD
 * Cloud
 * Databricks
 * DevOps
 * Docker
 * ETL
 * GCP
 * Support
 * Kafka
 * Kubernetes
 * Machine Learning
 * Python
 * PySpark
 * SQL
 * Spark