Lead Cloud Data Platform Engineer (AI & Data Engineering)

We are seeking a Lead Cloud Data Platform Engineer to build and modernize the Cyber Security Data Ecosystem on a hybrid cloud platform. This role combines advanced data engineering expertise with emerging AI technologies to develop scalable data products, automate data operations, and drive cloud transformation initiatives.

The ideal candidate will have deep experience in cloud-native data platforms, real-time data processing, AI-powered data solutions, and modern Lakehouse architectures. You will play a key role in designing and implementing next-generation analytics capabilities while helping guide the migration from on-premises environments to cloud-based platforms.

This position offers significant visibility and collaboration with principal engineers, product managers, architects, and data engineering teams across the organization., * Design, develop, and operationalize AI-enabled data platforms and data products on Google Cloud.

• Build scalable data ingestion, transformation, and distribution pipelines supporting large-scale analytics and cybersecurity initiatives.
• Utilize AI and agentic frameworks to automate data management, governance, quality monitoring, metadata management, and compliance processes.
• Develop and maintain real-time and batch data processing solutions using modern streaming technologies.
• Lead implementation of Lakehouse architectures and cloud-native data platforms.
• Partner with engineers, architects, and business stakeholders to define technical roadmaps and prioritize strategic data initiatives.
• Drive adoption of modern engineering standards, best practices, and emerging technologies across the data engineering organization.
• Support cloud migration efforts from on-premises environments to Google Cloud-based architecture.
• Mentor team members and provide technical leadership across multiple projects and initiatives.
• Ensure solutions are secure, scalable, reliable, and aligned with enterprise data governance requirements., * Spark
• Lakehouse Architecture
• Data Pipelines
• Data Modeling
• Data Governance
• Metadata Management

Streaming Technologies

• Apache Kafka
• Apache Flink
• Spark Streaming

AI & Machine Learning

• LangChain
• LangGraph
• Agent Development Kit (ADK)
• Agentic Frameworks
• RAG
• GraphRAG
• MCP

Why Join This Opportunity?

• Work on cutting-edge AI and data engineering initiatives within a large-scale enterprise environment.
• Drive innovation in cloud modernization and data platform transformation.
• Build intelligent data solutions that improve cybersecurity operations and analytics.
• Collaborate with highly skilled engineers and technology leaders.
• Gain exposure to advanced AI, cloud, and real-time data technologies at enterprise scale.

• Recent hands-on experience building AI-powered data solutions using:
• LangChain
• LangGraph or Agent Development Kit (ADK)
• Agentic AI frameworks
• Retrieval-Augmented Generation (RAG)
• GraphRAG
• Model Context Protocol (MCP)

Data Engineering Experience

• 5+ years of hands-on data engineering experience.
• Experience designing and supporting cloud-based data platforms and processing frameworks.
• Strong expertise building Spark-based ingestion and transformation solutions.

Cloud Data Platform Experience

• 3+ years of experience working with Data Lakehouse architectures and cloud-native data platforms.
• Hands-on experience with:
• Python
• PySpark
• Kafka
• Apache Airflow
• Google Cloud Storage (GCS)
• BigQuery
• Dataproc
• Cloud Composer

Streaming & Real-Time Data Processing

• Experience developing and maintaining real-time data processing solutions utilizing:
• Apache Kafka
• Apache Flink
• Spark Streaming

Preferred Qualifications

• Experience building AI-driven automation capabilities for enterprise data platforms.
• Knowledge of cybersecurity data ecosystems and analytics environments.
• Experience operating within Agile development teams.
• Strong understanding of cloud migration strategies and hybrid cloud architectures.
• Ability to influence technical direction and drive innovation across engineering organizations.

Technical Skills

Programming & Development

• Python
• PySpark

Cloud Platforms

• Google Cloud Platform (Google Cloud Platform)
• Google Cloud Storage
• BigQuery
• Dataproc
• Cloud Composer