Lead Cloud Data Engineer

We are seeking an experienced and forward-thinking Lead Cloud Data Engineer to drive the development of modern enterprise data platforms and AI-augmented solutions. In this role, you will be critical to supporting large-scale data warehousing operations, architecting centralized and federated data hubs, and spearheading enterprise data mesh modernization programs. This is a technical leadership role for an engineer who excels at designing distributed cloud architectures and enabling self-serve data capabilities. You will build end-to-end ELT pipelines, implement federated governance, integrate DevSecOps practices, and leverage generative AI frameworks to accelerate the software development lifecycle. If you are passionate about scaling modern data platforms and mentoring engineering teams, we want to hear from you., Data Engineering & Pipeline Development ETL/ELT Architecture: Develop and maintain robust end-to-end ingestion, transformation, and consumption-layer pipelines using advanced cloud data services. Data Quality & Governance: Implement automated data quality frameworks, anomaly detection, data lineage tracking, metadata management, and fine-grained access controls.

Data Mesh Architecture & Self-Serve Platforms Federated Governance: Design and implement a scalable Data Mesh architecture, establishing domain boundaries, data product specifications, and self-serve infrastructure patterns across modern analytics platforms (e.g., Databricks, Starburst, Collibra, Immuta). Domain Enablement: Build reusable frameworks, accelerators, and self-serve tools that empower cross-functional domain teams to independently publish, discover, and consume high-quality data products efficiently.

DevSecOps & AI-Augmented Development Infrastructure Automation: Integrate DevSecOps practices, including CI/CD automation, Infrastructure as Code (Terraform/CloudFormation), security scanning, and disaster recovery planning. AI Integration: Design and build agentic AI solutions, RAG pipelines, and intelligent automation using modern LLMs and AI-powered development tools to enhance platform capabilities and streamline workflows., AllSTEM Connections participates in the E-Verify program in certain locations as required by law. Learn more about the E-Verify program. _Participation_Poster_ES.pdf

We also consider for employment qualified applicants regardless of criminal histories, consistent with legal requirements, including, if applicable, the City of Los Angeles’ Fair Chance Initiative for Hiring Ordinance. Pursuant to applicable state and municipal Fair Chance Laws and Ordinances, we will consider for employment-qualified applicants with arrest and conviction records, including, if applicable, the San Francisco Fair Chance Ordinance. For Los Angeles, CA applicants: Qualified applications with arrest or conviction records will be considered for employment in accordance with the Los Angeles County Fair Chance Ordinance for Employers and the California Fair Chance Act. Pursuant to the San Francisco Fair Chance Ordinance, we will consider for employment qualified applicants with arrest and conviction records.

Experience Baseline: 5+ years of specialized experience designing and deploying distributed data architectures and modern cloud platforms. Cloud & Data Architecture: Deep, hands-on experience building scalable data solutions on major cloud services (e.g., AWS S3, Glue, Lake Formation, EMR, Redshift/Snowflake). Data Mesh & Governance: Demonstrated implementation of data mesh principles, including domain-oriented data ownership, data-as-a-product, and federated computational governance. Modern Tooling Proficiency: High familiarity with modern data stacks, including Databricks (Delta Lake, Unity Catalog), federated query engines (Starburst/Trino), and governance tools (Collibra/Immuta). DevSecOps & IaC: Strong background in CI/CD pipelines, containerization (Docker), and infrastructure automation (Terraform, CloudFormation). AI & Engineering: Practical experience leveraging generative AI tools for code acceleration, testing, and designing intelligent automation solutions.

Preferred Attributes Proven track record leading technical design sessions, architecture reviews, and mentoring data engineering teams. Excellent communication skills, with the ability to bridge complex technical concepts for both technical and executive stakeholders. Experience navigating hybrid cloud or secure enterprise environments.