> Markdown version of [/jobs/ext/2238159-manager-big-data-engineering-databricks-lead-hybrid](https://www.wearedevelopers.com/jobs/ext/2238159-manager-big-data-engineering-databricks-lead-hybrid). Every page supports `.md` or `Accept: text/markdown`. Links point to the HTML versions so they work for humans too. Agent guide: [/agents.md](https://www.wearedevelopers.com/agents.md). --- # Manager Big Data Engineering - Databricks Lead - Hybrid - **Company:** Sapient Corporation - **Location:** Atlanta, GA, United States - **Experience:** Expert - **Salary:** $130,000.0 - $180,000.0 - **Contract:** Permanent contract - **Skills:** Agile Methodology, Artificial Intelligence, Amazon Web Services, Data Analysis, Architectural Patterns, Automated Storage and Retrieval Systems, Automation of Tests, Microsoft Azure, Big Data, BigTable, BigQuery, Cloud Computing, Continuous Integration, Data as a Services, Data Architecture, Information Engineering, Data Infrastructure, Extract Transform Load (ETL), Data Systems, Data Warehousing, Dimensional Modeling, Distributed Computing Environment, Distributed Data Store, Amazon DynamoDB, Data Flow Control, Apache Hive, Python (Programming Language), Machine Learning, NoSQL, Operational Databases, Parsing, Performance Tuning, Release Management, Cloud Services, Azure Machine Learning, Search Technologies, Software Engineering, Data Streaming, Google Cloud, Azure Data Factory, Large Language Models, Snowflake, Apache Spark, Generative AI, Containerization, Data Lakes, AI Platforms, Pyspark, Kubernetes, Infrastructure Automation Frameworks, Cosmos DB, Spark Streaming, Data Management, Machine Learning Operations, Vertica, Functional Programming, Software Version Control, Data Pipelines, Amazon Redshift, Databricks - **Published:** August 26, 2026 - **Apply:** https://jobs.localjobnetwork.com/apply/add/88148112/1 ## About the Role Your Skills and Experience * 10+ years of demonstrated experience leading the implementation of enterprise-scale data platforms and end-to-end data engineering solutions in production environments. * Hands-on experience with Databricks as a primary data engineering platform, including Delta Lake, Databricks Workflows, Databricks SQL, notebooks, jobs, and modern Lakehouse architecture patterns. * Strong experience designing and implementing scalable data platforms on one or more public cloud platforms including Amazon Web Services (AWS), Microsoft Azure, or Google Cloud Platform (GCP). * Advanced proficiency in Python and practical experience using Python-based frameworks for data engineering, platform automation, and AI-enabled engineering workflows. * Strong expertise in Apache Spark, PySpark, Spark SQL, and distributed data processing technologies. * Experience implementing both batch and real-time data pipelines using technologies such as Spark Streaming, Glue ETL, Lambda, Dataflow, Azure Data Factory, Databricks, or similar frameworks. * Experience with data modeling, dimensional modeling, data warehousing, and modern architectural patterns including Lakehouse and data mesh approaches. * Experience with columnar data platforms such as Snowflake, BigQuery, Redshift, Vertica, or similar technologies. * Experience with NoSQL technologies such as DynamoDB, Bigtable, Cosmos DB, or equivalent distributed databases. * Experience implementing software engineering best practices including source control, CI/CD, automated testing, release management, infrastructure automation, and production support processes. * Familiarity with MLOps concepts and supporting data engineering responsibilities related to model deployment, validation, monitoring, rollback, governance, and operational reliability. * Experience leading engineering teams, managing delivery workstreams, and collaborating effectively across cross-functional and client-facing environments. * Strong communication, stakeholder management, and consulting skills. AI Engineering & Modern Data Platform Experience * Experience supporting AI-enabled solutions through the design and implementation of scalable, production-grade data platforms. * Exposure to AI engineering concepts including context engineering, retrieval-augmented generation (RAG), agentic architectures, semantic search, and production data services supporting AI-powered experiences. * Experience building and maintaining the data pipelines that support retrieval systems, including document ingestion, parsing, chunking, metadata extraction, embedding generation, and incremental indexing processes. * Familiarity with vector databases, semantic search platforms, graph-based knowledge stores, and modern retrieval architectures. * Exposure to cloud AI services such as Vertex AI, Azure AI Services, AWS AI Services, or similar AI platforms. * Experience supporting AI and machine learning lifecycle requirements, including evaluation datasets, monitoring, operational telemetry, validation workflows, release management, and platform observability. * Understanding of platform requirements for managing agent state, conversation history, session context, memory stores, and durable retrieval structures that support AI-enabled applications. * Ability to apply enterprise data engineering principles such as lineage, governance, provenance, observability, and data contracts to AI-enabled platforms and retrieval systems. * Experience supporting agentic frameworks, orchestration platforms, or emerging AI engineering technologies is a plus. * Experience with Snowflake and zero-copy architecture patterns is a plus, particularly within retail, financial services, energy, logistics, manufacturing, or CPG industries. Set Yourself Apart With * Databricks Data Engineer Associate, Professional, or Machine Learning certifications. * Certifications in AWS, Microsoft Azure, Google Cloud, Snowflake, or related cloud and data technologies. * Experience leading Databricks-based modernization initiatives and enterprise-scale Lakehouse implementations. * Demonstrated experience applying AI engineering concepts and Generative AI technologies in production business environments. * Hands-on experience supporting AI/ML and LLM lifecycle requirements, including deployment support, monitoring, validation, evaluation infrastructure, and operational governance. * Experience in retail, financial services, energy, manufacturing, logistics, healthcare, CPG, or other data-intensive industries. * Experience working in consulting, digital transformation, or client-facing delivery environments. * Understanding of Agile, product, and modern delivery methodologies. ## Description Manager Data Engineering, Databricks Lead Publicis Sapient is seeking a Manager, Data Engineering with deep Databricks expertise to lead the design, delivery, and modernization of enterprise-scale data platforms. This role combines hands-on technical leadership, client engagement, architecture ownership, and team management. You will help clients build modern Lakehouse architectures, scalable data products, and AI-ready data foundations using Databricks and cloud-native technologies. The source role emphasizes Databricks, Python, cloud data platforms, AI engineering, and modern data architectures. Your Impact * Combine your technical expertise, leadership skills, and problem-solving passion to work closely with clients, translating complex business challenges into modern data platform solutions that deliver measurable business value. * Lead the architecture, design, and delivery of enterprise-scale data engineering solutions built on Databricks and cloud-native data platforms. * Drive data modernization initiatives, helping clients migrate from traditional data architectures to modern lakehouse and cloud-based ecosystems.Lead the development and optimization of batch and streaming data pipelines using Databricks, Spark, and cloud-native data services. * Design scalable data foundations that support analytics, machine learning, Generative AI, and AI-enabled experiences through high-quality data products and services.Partner with stakeholders to define data platform roadmaps, architecture standards, governance practices, and delivery approaches. * Establish best practices for engineering excellence, performance optimization, data quality, observability, reliability, security, and operational support. * Support AI-enabled engineering use cases by designing scalable retrieval patterns, context engineering approaches, and modern data services that power machine learning and agentic solutions. * Conduct technical feasibility assessments, project estimation, architecture reviews, and solution planning activities for large-scale client engagements. * Mentor and develop engineers while providing technical leadership, delivery oversight, and career guidance across multiple project teams. * Contribute to practice growth through client engagement, solution development, capability building, hiring, and thought leadership. ## Related Videos - [PySpark - Combining Machine Learning & Big Data](https://www.wearedevelopers.com/videos/44-pyspark-combining-machine-learning-big-data) - [From Messy Queries to Scalable Systems - How Data Engineering actually works](https://www.wearedevelopers.com/videos/100203-from-messy-queries-to-scalable-systems-how-data-engineering-actually-works) - [Leveraging Real time data in FSIs](https://www.wearedevelopers.com/videos/806-leveraging-real-time-data-in-fsis) - [Data Science in Retail](https://www.wearedevelopers.com/videos/586-data-science-in-retail) - [Empowering Retail Through Applied Machine Learning](https://www.wearedevelopers.com/videos/976-empowering-retail-through-applied-machine-learning) - [Modern Data Architectures need Software Engineering](https://www.wearedevelopers.com/videos/1030-modern-data-architectures-need-software-engineering) ## Related Articles - [How to Become an AI Engineer](https://www.wearedevelopers.com/magazine/331-how-to-become-an-ai-engineer) - [Top Big Data Technologies That You Need to Know](https://www.wearedevelopers.com/magazine/108-top-big-data-technologies-that-you-need-to-know) - [Navigating the AI Shift](https://www.wearedevelopers.com/magazine/629-navigating-the-ai-shift) - [Got AI ideas but no money? Here are 10 free ways to level up your AI skills with Google Cloud](https://www.wearedevelopers.com/magazine/600-got-ai-ideas-but-no-money-here-are-10-free-ways-to-level-up-your-ai-skills-with-google-cloud) - [Graph and AI Trends 2026: Why Is AI Running but Not Yet Delivering?](https://www.wearedevelopers.com/magazine/680-graph-and-ai-trends-2026-why-is-ai-running-but-not-yet-delivering) - [Making Data Warehouses Fast: A Developer’s Story](https://www.wearedevelopers.com/magazine/107-making-data-warehouses-fast-a-developer-s-story)