> Markdown version of [/jobs/ext/2149089-lead-cloud-data-engineer](https://www.wearedevelopers.com/jobs/ext/2149089-lead-cloud-data-engineer). Every page supports `.md` or `Accept: text/markdown`. Links point to the HTML versions so they work for humans too. Agent guide: [/agents.md](https://www.wearedevelopers.com/agents.md). --- # Lead Cloud Data Engineer - **Company:** Akraya, Inc. - **Location:** San Francisco, CA, United States - **Experience:** Expert - **Salary:** $187,200.0 - $208,000.0 - **Contract:** Permanent contract - **Skills:** Unity 3d, Artificial Intelligence, Amazon Web Services, Amazon S3, Business Analytics Applications, Data Analysis, Big Data, Cloud Computing, Cloud Database, Software Documentation, Cyber Security, Information Systems, Databases, Continuous Delivery, Continuous Integration, Information Engineering, Data Governance, Data Hub, Data Infrastructure, Extract Transform Load (ETL), Data Warehousing, Github, Python (Programming Language), Meta-Data Management, Systems Development Life Cycle, Cloud Services, Standard Sql, Requirements Management, Software Engineering, SQL Databases, Scripting, Cloud Platform System, Large Language Models, Electronic Medical Records, AWS Lambda, HybridCloud, Cloudformation, Containerization, Data Lakes, Gitlab-ci, Information Technology, Data Lineage, Collibra, Data Analytics, AWS Data Analytics, Data Management, Cloud Migration, Terraform, Domain Driven Design, Data Pipelines, Devsecops, Cisco, Docker, Jenkins, Databricks - **Published:** August 20, 2026 - **Apply:** https://www.careerbuilder.com/job-details/lead-cloud-data-engineer-26-02136-san-francisco-ca--cad106ff-72b3-4845-85a2-fa7c7c0e8f2a ## About the Role * 8+ years of experience in Data Engineering with at least 5+ years designing enterprise AWS Data Platforms. * Strong hands-on experience with AWS S3, Glue, EMR, Redshift, Lambda, Kinesis, Lake Formation, and Step Functions. * Proven experience implementing Data Mesh architecture, domain-driven data products, and federated governance. * Expertise with Databricks (Delta Lake, Unity Catalog), Starburst/Trino, Collibra, and Immuta. * Strong knowledge of ETL/ELT development, data modeling, metadata management, data lineage, and data quality frameworks. * Experience with Terraform, CloudFormation, Docker, ECS, GitHub Actions/GitLab CI, Jenkins, and DevSecOps best practices. * Experience implementing AI-enabled engineering solutions using Amazon Bedrock, LLMs, RAG architectures, Copilot, Claude Code, or similar AI frameworks. * Strong SQL, Python, and cloud-native data engineering expertise. * Experience supporting secure, compliant cloud environments, including AWS GovCloud or regulated environments. * Excellent leadership, mentoring, architecture design, communication, and stakeholder management skills. Nice-to-have Skills: * Experience supporting enterprise data warehouse modernization initiatives. * Knowledge of hybrid cloud and on-premises data platform integration. * Experience implementing self-service analytics and reusable data product frameworks. * Familiarity with cost optimization strategies for large-scale cloud data platforms. * Experience working within highly regulated industries such as Financial Services or Government. Preferred Qualifications: * Bachelor's or Master's degree in Computer Science, Data Engineering, Information Systems, or a related technical field. * AWS Certifications (Solutions Architect, Data Analytics, or Developer) preferred. * Experience leading enterprise-scale cloud modernization and AI transformation initiatives. * Strong background in enterprise architecture, technical leadership, and cross-functional collaboration., AWS Lambda, Amazon Simple Storage Service (S3), Amazon Web Services (AWS), Architectural Analysis, Architectural Design, Artificial Intelligence (AI), Automation, Best Practices, Cisco Unity, Cloud Computing, Computer Science, Continuous Deployment/Delivery, Continuous Integration, Cost Control, Cross-Functional, Data Analysis, Data Warehousing, Database Extract Transform and Load (ETL), Docker, Documentation Standards, Electronic Medical Records, Enterprise Architecture, Finance, Financial Services, GitHub, Government, Hybrid Cloud, Information Technology & Information Systems, Jenkins, Leadership, Mentoring, Production Support, Python Programming/Scripting Language, Requirements Management, SQL (Structured Query Language), Scalable System Development, Security Infrastructure, Software Development Lifecycle (SDLC), Software Engineering, Technical Leadership, Technical Recruiting, Technical Writing ## Description We are seeking a Lead Cloud Data Engineer - Data Mesh & AI to lead the design, development, and modernization of enterprise-scale data platforms supporting large data warehouse and data hub initiatives. The ideal candidate will have deep expertise in AWS data engineering, Data Mesh architecture, Databricks, DevSecOps, and AI-driven engineering practices. This role will drive the implementation of scalable data products, federated governance, and AI-augmented solutions while providing technical leadership across cross-functional engineering teams., * Design, develop, and maintain scalable ETL/ELT pipelines using AWS services including Glue, EMR, Lambda, Kinesis, Step Functions, Redshift, and S3. * Architect and implement enterprise Data Mesh solutions with domain-oriented ownership, federated governance, and reusable self-service data product frameworks. * Develop modern data platforms leveraging Databricks, Delta Lake, Unity Catalog, Starburst/Trino, Collibra, and Immuta. * Implement enterprise data governance, metadata management, data lineage, access controls, and automated data quality monitoring frameworks. * Build and maintain CI/CD pipelines, Infrastructure as Code (Terraform/CloudFormation), containerized deployments, and DevSecOps automation. * Design and implement AI-powered solutions including RAG pipelines, agentic AI workflows, Amazon Bedrock, LLM integrations, and AI-assisted SDLC automation. * Optimize cloud infrastructure for scalability, security, performance, and cost efficiency across hybrid cloud environments. * Provide L3 production support, mentor engineering teams, and establish best practices for modern data engineering and Data Mesh adoption. * Lead technical architecture reviews, collaborate with business stakeholders, and translate business requirements into scalable technical solutions. * Create and maintain technical architecture documentation, implementation standards, and engineering best practices. ## Related Videos - [From Messy Queries to Scalable Systems - How Data Engineering actually works](https://www.wearedevelopers.com/videos/100203-from-messy-queries-to-scalable-systems-how-data-engineering-actually-works) - [Docker Compose: Rediscovered](https://www.wearedevelopers.com/videos/1978-docker-compose-rediscovered) - [Innovating Developer Tools with AI: Insights from GitHub Next](https://www.wearedevelopers.com/videos/1268-innovating-developer-tools-with-ai-insights-from-github-next) - [How Cisco embraced a DevOps culture within its network engineering team](https://www.wearedevelopers.com/videos/99-how-cisco-embraced-a-devops-culture-within-its-network-engineering-team) - [Docker build without Docker](https://www.wearedevelopers.com/videos/100114-docker-build-without-docker) - [Bringing AI Model Testing and Prompt Management to Your Codebase with GitHub Models](https://www.wearedevelopers.com/videos/1536-bringing-ai-model-testing-and-prompt-management-to-your-codebase-with-github-models) ## Related Articles - [7 Cloud Computing Trends Coming in 2025 for Developers](https://www.wearedevelopers.com/magazine/412-7-cloud-computing-trends-coming-in-2025-for-developers) - [What Are The Top Skills Required For Azure Developers?](https://www.wearedevelopers.com/magazine/77-what-are-the-top-skills-required-for-azure-developers) - [Highest Paying Tech Companies for Developers](https://www.wearedevelopers.com/magazine/220-highest-paying-tech-companies-for-developers) - [Making Data Warehouses Fast: A Developer’s Story](https://www.wearedevelopers.com/magazine/107-making-data-warehouses-fast-a-developer-s-story) - [Got AI ideas but no money? Here are 10 free ways to level up your AI skills with Google Cloud](https://www.wearedevelopers.com/magazine/600-got-ai-ideas-but-no-money-here-are-10-free-ways-to-level-up-your-ai-skills-with-google-cloud) - [Data Engineer Salary UK](https://www.wearedevelopers.com/magazine/253-data-engineer-salary-uk)