Lead Big Data Engineer

We are seeking a highly skilled Lead Big Data Engineer to join our team and work closely with our client in designing, building, and modernizing enterprise-scale data platforms. The ideal candidate will have deep expertise in streaming architectures, event-driven systems, real-time data processing, and modern cloud-based data platforms. This role requires a strong technical leader who can drive architecture decisions, mentor engineering teams, and deliver scalable, secure, and high-performance data solutions., Leadership & Architecture

• Define and drive data platform architecture, engineering standards, and best practices.
• Lead and mentor a team of data engineers, providing technical guidance and support.
• Drive technical decision-making and ensure alignment with business objectives.
• Collaborate with Architects, Product Owners, and cross-functional engineering teams.

Data Platform Engineering

• Design and implement end-to-end data ingestion frameworks.
• Build and optimize real-time and batch data pipelines.
• Develop scalable event-driven architectures using Kafka and related technologies.
• Design and implement Change Data Capture (CDC) solutions for transactional data replication.
• Develop and maintain modern data platforms leveraging Databricks and Lakehouse architectures.
• Support cloud migration and data platform modernization initiatives.

Performance, Reliability & Governance

• Ensure solutions meet requirements for scalability, reliability, security, and governance.
• Implement data modeling and performance optimization strategies.
• Support monitoring, troubleshooting, and continuous improvement of data platforms.
• Enable data-driven decision-making through reliable and accessible data solutions., * Lead the design and evolution of a modern enterprise data platform.
• Work with cutting-edge technologies including Kafka, Confluent Cloud, Databricks, Spark, and Delta Lake.
• Drive real-time data processing and event-driven architecture initiatives.
• Influence technical strategy and architecture decisions.
• Collaborate with talented teams on large-scale cloud modernization programs.
• Make a direct impact on business outcomes through data-driven innovation.

• 12+ years of experience in Data Engineering, Big Data Engineering, or Data Platform Engineering.
• Proven experience leading engineering teams and driving large-scale data initiatives.
• Strong experience designing both streaming and batch data processing systems.

Technical Skills Streaming & Event-Driven Systems

• Strong expertise in streaming data architectures and event-driven systems.

• Hands-on experience with:

• Apache Kafka
• Confluent Platform / Confluent Cloud

Deep understanding of:

• Kafka Topics and Partitions
• Schema Design
• Schema Registry
• Kafka Connect
• ksqlDB

Data Processing & Lakehouse Platforms

• Strong hands-on experience with:

• Databricks
• Apache Spark (PySpark and/or Scala)
• Delta Lake

Experience designing and implementing Lakehouse architectures.

Programming & Query Languages

• Advanced proficiency in:

• Python
• SQL

Scala experience is a plus.

CDC & Data Integration

• Strong understanding of Change Data Capture (CDC) patterns and implementation.

Cloud Platforms

• Experience with one or more major cloud providers:

• Microsoft Azure
• Amazon Web Services (AWS)
• Google Cloud Platform (Google Cloud Platform)

Data Engineering Best Practices

• Data modeling
• Performance tuning and optimization
• Security and governance controls, * Debezium
• Kafka Connect ecosystem
• Oracle GoldenGate
• SQL-based CDC tools

Experience with orchestration tools:

• Apache Airflow
• Databricks Workflows

Infrastructure as Code (IaC):

• Terraform

CI/CD pipelines for data platforms.

Experience with monitoring and observability platforms.

Exposure to multi-cloud data platform strategies.