> Markdown version of [/jobs/ext/2359885-gcp-data-lead](https://www.wearedevelopers.com/jobs/ext/2359885-gcp-data-lead). Every page supports `.md` or `Accept: text/markdown`. Links point to the HTML versions so they work for humans too. Agent guide: [/agents.md](https://www.wearedevelopers.com/agents.md). --- # GCP Data Lead - **Company:** HCLTech - **Location:** West Sussex, UK - **Experience:** Expert - **Contract:** Permanent contract - **Skills:** Java (Programming Language), Adaptable Database Systems, Airflow, Amazon Web Services, Application Frameworks, Automation of Tests, Microsoft Azure, Big Data, BigQuery, Cloud Database, Computer Programming, Databases, Continuous Integration, Data Governance, Extract Transform Load (ETL), Dataspaces, Data Systems, Shard (Database Architecture), IBM InfoSphere DataStage, DevOps, Distributed Data Store, Distributed Systems, Apache HBase, Identity and Access Management, Python (Programming Language), Enterprise Messaging Systems, MongoDB, NoSQL, Performance Tuning, SQL Databases, Data Streaming, Data Ingestion, Snowflake, Apache Spark, Containerization, Kubernetes, Apache Flink, Apache Kafka, Data Pipelines, Amazon Redshift - **Published:** August 27, 2026 - **Apply:** https://uk.indeed.com/viewjob?jk=9b884af838be109c ## About the Role Strong programming skills in Java (preferred) , Python , or Scala . Hands-on experience with cloud data services (GCP preferred; Azure/AWS acceptable). Practical experience with distributed data processing frameworks such as Spark (Core/SQL/Streaming), Flink, or Storm. Experience across multiple database technologies-Relational, NoSQL, MPP, columnar. Strong knowledge of data ingestion, transformation and messaging systems : Kafka, Pulsar, Pub/Sub, etc. Understanding of designing scalable data models for varied access patterns. Experience with performance tuning , cost-optimisation and scaling strategies. Experience delivering large-scale big data solutions in batch and/or streaming environments, on cloud or on-premise. Good familiarity with the wider data ecosystem and open-source frameworks. Experience with orchestration (Airflow/Composer) and workflow automation. Understanding of DevOps for data systems: CI/CD, containers, Kubernetes and automated testing. Knowledge of security for big-data systems including IAM, encryption, and cluster-level controls. Basic understanding of monitoring and alerting for distributed systems. Knowledge of dimensional modelling (star, snowflake, normalized/denormalized). Awareness of data governance, cataloguing and lineage tools. ## Description The IDEA Lab is expanding its GCP Data Products capability and is looking for a highly skilled Data Engineer to build, optimise, and scale data pipelines and large-scale data processing workloads in a cloud-native environment. You will work with modern distributed data systems, contribute to data platform modernisation, and support large-scale ingestion, transformation, and analytics workloads across the Business Transaction Banking platform., Design and deliver end-to-end data pipelines on cloud platforms (GCP preferred). Build scalable data ingestion, transformation, and processing workflows using distributed technologies such as Spark, Flink, Storm, or similar. Develop robust ELT/ETL processing and migration pipelines, including support for legacy Datastage decommissioning and modernisation. Work with a variety of database technologies including relational, NoSQL, MPP and columnar stores (BigQuery, Redshift, Azure SQLDW, HBase, MongoDB). Implement streaming and messaging-based pipelines using Kafka, Pulsar or Pub/Sub. Build optimised, scalable data models to support diverse consumption patterns, applying partitioning, sharding, bucketing and aggregation strategies. Apply performance tuning and optimisation across storage, compute and query layers. Ensure secure handling of data including authentication, authorisation, encryption in transit/at rest, and cloud-native security controls. Implement monitoring, alerting and observability for large-scale distributed data workloads. Use orchestration tools such as Cloud Composer, Airflow or equivalent to operationalise pipelines. Contribute to CI/CD, containerisation, Kubernetes-based deployments, and automated testing practices. Participate in data governance, metadata, catalogue and lineage processes as needed. Collaborate with engineers, architects and SMEs to deliver stable, high-quality data products. ## Related Videos - [From Messy Queries to Scalable Systems - How Data Engineering actually works](https://www.wearedevelopers.com/videos/100203-from-messy-queries-to-scalable-systems-how-data-engineering-actually-works) - [Alibaba Big Data and Machine Learning Technology](https://www.wearedevelopers.com/videos/37-alibaba-big-data-and-machine-learning-technology) - [From DevOps to Scaled DevOps: How We’re Rebuilding Continuous Delivery as a Platform](https://www.wearedevelopers.com/videos/100018-from-devops-to-scaled-devops-how-we-re-rebuilding-continuous-delivery-as-a-platform) - [Leveraging Real time data in FSIs](https://www.wearedevelopers.com/videos/806-leveraging-real-time-data-in-fsis) - [Enjoying SQL data pipelines with dbt](https://www.wearedevelopers.com/videos/823-enjoying-sql-data-pipelines-with-dbt) - [Tomorrow's cloud data platforms - fully managed database-as-a-service (DBaaS)](https://www.wearedevelopers.com/videos/254-tomorrow-s-cloud-data-platforms-fully-managed-database-as-a-service-dbaas) ## Related Articles - [Top Big Data Technologies That You Need to Know](https://www.wearedevelopers.com/magazine/108-top-big-data-technologies-that-you-need-to-know) - [Making Data Warehouses Fast: A Developer’s Story](https://www.wearedevelopers.com/magazine/107-making-data-warehouses-fast-a-developer-s-story) - [7 Cloud Computing Trends Coming in 2025 for Developers](https://www.wearedevelopers.com/magazine/412-7-cloud-computing-trends-coming-in-2025-for-developers) - [Data Engineer Salary UK](https://www.wearedevelopers.com/magazine/253-data-engineer-salary-uk) - [How We Built a Worry-Free System That Runs for 10+ Years – And What We’d Do Again](https://www.wearedevelopers.com/magazine/751-how-we-built-a-worry-free-system-that-runs-for-10-years-and-what-we-d-do-again) - [Got AI ideas but no money? Here are 10 free ways to level up your AI skills with Google Cloud](https://www.wearedevelopers.com/magazine/600-got-ai-ideas-but-no-money-here-are-10-free-ways-to-level-up-your-ai-skills-with-google-cloud)