Lead Data Engineer

Ecloud Labs
yesterday

Role details

Contract type
Permanent contract
Employment type
Full-time (> 32 hours)
Working hours
Regular working hours
Languages
English
Experience level
Senior

Job location

Tech stack

Java
Artificial Intelligence
Airflow
Amazon Web Services (AWS)
Data analysis
Azure
Big Data
Computer Programming
Continuous Integration
Data Systems
Python
Data Streaming
Google Cloud Platform
Large Language Models
Spark
Containerization
Data Lake
Kubernetes
Infrastructure Automation Frameworks
Apache Flink
Data Analytics
Real Time Data
Kafka
Data Management

Job description

We are seeking a Lead Data Engineer to play a critical role in big data engineering and analytics capabilities. This role will lead the design, development, and optimization of scalable data platforms, real time data pipelines, and analytics ready data products.

As part of the Big Data Engineering and Analytics organization, this role will focus on enabling a modern data lake and data mesh architecture that supports high volume, high velocity, and high variety data across enterprise and platform domains.

The Lead Data Engineer will bring deep hands-on expertise in Kubernetes and Kafka streaming platforms. This role partners closely with data architects, analytics teams, and platform engineering to deliver reliable, governed, and high-quality data solutions.

Responsibilities

  • Lead the design and implementation of scalable big data pipelines for batch and real time processing

  • Build and operate streaming data platforms using Kafka

  • Design and deploy cloud native data solutions across AWS, Azure, and Google Cloud Platform

  • Develop and manage containerized workloads using Kubernetes

  • Enable data mesh architecture with domain oriented data products

  • Design and implement data lake and lakehouse architectures

  • Ensure data quality, reliability, and observability

  • Implement governance capabilities including metadata and lineage

  • Collaborate with analytics, AI, and business teams

  • Optimize performance and cost efficiency

Requirements

13+ years of experience in big data engineering

  • Strong experience with Kafka

  • Strong experience with Kubernetes

  • Experience with AWS

  • Experience building data lakes or lakehouse platforms

  • Experience with data mesh concepts

  • Strong programming skills in Python, Scala, or Java

  • Experience with Spark, Flink, or Beam

  • Experience with Airflow or orchestration tools

  • Understanding of data modeling and governance

  • Experience with CICD and infrastructure automation

  • Experience supporting AI and machine learning workloads including large language models such as Claude or similar platforms

  • Strong communication and leadership skills

Apply for this position