Lead Cloud Data Platform Engineer (AI & Data Engineering)

THE JUDGE GROUP, INC.
Chandler, United States of America
yesterday

Role details

Contract type
Permanent contract
Employment type
Full-time (> 32 hours)
Working hours
Regular working hours
Languages
English
Experience level
Senior
Compensation
$ 154K

Job location

Chandler, United States of America

Tech stack

Agile Methodologies
Artificial Intelligence
Airflow
Google BigQuery
Cloud Computing
Computer Security
Computer Programming
Data Architecture
Information Engineering
Data Governance
Dataspaces
Data Systems
Python
Meta-Data Management
Cloud Services
DataOps
Cloudera
Data Streaming
Enterprise Data Management
Data Processing
Google Cloud Platform
Cloud Platform System
Data Ingestion
Spark
Generative AI
HybridCloud
PySpark
Apache Flink
Data Analytics
Real Time Data
Kafka
Spark Streaming
Data Management
Data Lakehouse
Video Streaming
Cloud Migration
Virtual Agents
Stream Processing
Data Pipelines

Job description

We are seeking a Lead Cloud Data Platform Engineer to build and modernize the Cyber Security Data Ecosystem on a hybrid cloud platform. This role combines advanced data engineering expertise with emerging AI technologies to develop scalable data products, automate data operations, and drive cloud transformation initiatives.

The ideal candidate will have deep experience in cloud-native data platforms, real-time data processing, AI-powered data solutions, and modern Lakehouse architectures. You will play a key role in designing and implementing next-generation analytics capabilities while helping guide the migration from on-premises environments to cloud-based platforms.

This position offers significant visibility and collaboration with principal engineers, product managers, architects, and data engineering teams across the organization., * Design, develop, and operationalize AI-enabled data platforms and data products on Google Cloud.

  • Build scalable data ingestion, transformation, and distribution pipelines supporting large-scale analytics and cybersecurity initiatives.
  • Utilize AI and agentic frameworks to automate data management, governance, quality monitoring, metadata management, and compliance processes.
  • Develop and maintain real-time and batch data processing solutions using modern streaming technologies.
  • Lead implementation of Lakehouse architectures and cloud-native data platforms.
  • Partner with engineers, architects, and business stakeholders to define technical roadmaps and prioritize strategic data initiatives.
  • Drive adoption of modern engineering standards, best practices, and emerging technologies across the data engineering organization.
  • Support cloud migration efforts from on-premises environments to Google Cloud-based architecture.
  • Mentor team members and provide technical leadership across multiple projects and initiatives.
  • Ensure solutions are secure, scalable, reliable, and aligned with enterprise data governance requirements., * Spark
  • Lakehouse Architecture
  • Data Pipelines
  • Data Modeling
  • Data Governance
  • Metadata Management

Streaming Technologies

  • Apache Kafka
  • Apache Flink
  • Spark Streaming

AI & Machine Learning

  • LangChain
  • LangGraph
  • Agent Development Kit (ADK)
  • Agentic Frameworks
  • RAG
  • GraphRAG
  • MCP

Why Join This Opportunity?

  • Work on cutting-edge AI and data engineering initiatives within a large-scale enterprise environment.
  • Drive innovation in cloud modernization and data platform transformation.
  • Build intelligent data solutions that improve cybersecurity operations and analytics.
  • Collaborate with highly skilled engineers and technology leaders.
  • Gain exposure to advanced AI, cloud, and real-time data technologies at enterprise scale.

Requirements

  • Recent hands-on experience building AI-powered data solutions using:
  • LangChain
  • LangGraph or Agent Development Kit (ADK)
  • Agentic AI frameworks
  • Retrieval-Augmented Generation (RAG)
  • GraphRAG
  • Model Context Protocol (MCP)

Data Engineering Experience

  • 5+ years of hands-on data engineering experience.
  • Experience designing and supporting cloud-based data platforms and processing frameworks.
  • Strong expertise building Spark-based ingestion and transformation solutions.

Cloud Data Platform Experience

  • 3+ years of experience working with Data Lakehouse architectures and cloud-native data platforms.
  • Hands-on experience with:
  • Python
  • PySpark
  • Kafka
  • Apache Airflow
  • Google Cloud Storage (GCS)
  • BigQuery
  • Dataproc
  • Cloud Composer

Streaming & Real-Time Data Processing

  • Experience developing and maintaining real-time data processing solutions utilizing:
  • Apache Kafka
  • Apache Flink
  • Spark Streaming

Preferred Qualifications

  • Experience building AI-driven automation capabilities for enterprise data platforms.
  • Knowledge of cybersecurity data ecosystems and analytics environments.
  • Experience operating within Agile development teams.
  • Strong understanding of cloud migration strategies and hybrid cloud architectures.
  • Ability to influence technical direction and drive innovation across engineering organizations.

Technical Skills

Programming & Development

  • Python
  • PySpark

Cloud Platforms

  • Google Cloud Platform (Google Cloud Platform)
  • Google Cloud Storage
  • BigQuery
  • Dataproc
  • Cloud Composer

Apply for this position