> Markdown version of [/jobs/ext/4487195-lead-digital-data-engineer-databricks](https://www.wearedevelopers.com/jobs/ext/4487195-lead-digital-data-engineer-databricks). Every page supports `.md` or `Accept: text/markdown`. Links point to the HTML versions so they work for humans too. Agent guide: [/agents.md](https://www.wearedevelopers.com/agents.md).

---

# Lead Digital Data Engineer (Databricks)

- **Company:** BJ's Wholesale Club, Inc.
- **Location:** Marlborough, MA, United States
- **Experience:** Expert
- **Salary:** $128,500 - $164,000
- **Contract:** Permanent contract
- **Skills:** Adobe Analytics, Artificial Intelligence, Amazon Web Services, Amazon S3, Automation of Tests, Microsoft Azure, Continuous Integration, Data Architecture, Information Engineering, Data Governance, Digital Data, Distributed Data Store, Amazon DynamoDB, Google Analytics, Identity and Access Management, Operational Data Store, Scrum Methodology, Systems Development Life Cycle, Search Technologies, SQL Databases, Data Streaming, Systems Integration, Apache Spark, Data Lakes, Pyspark, Git Flow, Information Technology, Data Analytics, Apache Kafka, Machine Learning Operations, Functional Programming, Terraform, Software Version Control, Data Pipelines, Confluent, Amazon Redshift, Databricks
- **Published:** October 27, 2026
- **Apply:** https://www.indeed.com/viewjob?jk=37c50544c2bc1f59

## About the Role

Bachelor's degree in Computer Science, Information Technology, or equivalent, plus 7+ years of experience in data engineering, including 3+ years of hands-on Databricks experience in production. Experience must include:

 * Expert-level Apache Spark (PySpark and SQL), Delta Lake, and Lakehouse architecture
 * Unity Catalog, Databricks Workflows, and SQL Warehouses
 * Integrating AWS services: S3, Kafka (MSK/Confluent), DynamoDB, Lambda, and IAM
 * Building data quality and validation frameworks
 * Genie Spaces, AI/BI dashboards, and building Databricks agents or GenAI solutions (Mosaic AI, MLflow)
 * Building large-scale distributed data systems on AWS, Azure, or GCP
 * Working in an Agile Scrum environment and leading distributed teams
 * SDLC, CI/CD, version control, and test automation for data pipelines

## Description

As Lead Digital Data Engineer, you will own the design, build, and evolution of BJ's Databricks Lakehouse, working closely with product managers, program managers, and business unit heads. You will turn digital, fulfillment, and operational data into trusted, governed, self-service data products, and lead our adoption of AI-powered analytics, including Genie and Databricks agents.
Key Responsibilities

 * Design and maintain a scalable Lakehouse using Delta Lake and the medallion (Bronze/Silver/Gold) architecture, serving as the single source of truth for analytics following the migration from AWS Redshift.
 * Build batch and real-time pipelines integrating Kafka, DynamoDB, S3, and other AWS services with Databricks (Structured Streaming, Auto Loader, Delta Live Tables).
 * Ingest and integrate digital sources such as Adobe Analytics, AEP, and Google Analytics alongside transactional and operational data.
 * Implement data quality validation frameworks with automated checks, alerting, and SLAs so data is accurate and clearly represented across business verticals.
 * Lead data governance through Unity Catalog (access control, lineage, classification) and monitor pipeline health, freshness, and cost.
 * Design reusable Genie Spaces and AI/BI dashboards with certified metrics for business self-service.
 * Develop and productionize Databricks agents and AI use cases using Mosaic AI, MLflow, Model Serving, and Vector Search.
 * Optimize performance and cost across clusters, SQL Warehouses, and Delta tables.
 * Establish CI/CD for data pipelines (Databricks Asset Bundles, Git workflows, automated testing, Terraform).
 * Own and prioritize the digital data and analytics roadmap with product leaders.
 * Lead and mentor a distributed team of up to 10 engineers across onshore and offshore locations, driving Agile delivery and providing regular status and risk updates to senior leadership.