Senior Data Platform / Data Product Engineering Lead

We are looking for a Senior Data Platform / Data Product Engineering Lead to drive enterprise-scale data product lifecycle enablement across modern data platforms. This role will lead the design, standardization, and adoption of a paved path for data creators, enabling self-service, governed, and scalable data product development. The role requires deep expertise in Databricks, Airflow (Astronomer), CI/CD automation, data governance, and data marketplace constructs, along with the ability to lead platform transformation initiatives and mentor engineering teams., We are looking for a Senior Data Platform / Data Product Engineering Lead to drive enterprise-scale data product lifecycle enablement across modern data platforms. This role will lead the design, standardization, and adoption of a paved path for data creators, enabling self-service, governed, and scalable data product development. The role requires deep expertise in Databricks, Airflow (Astronomer), CI/CD automation, data governance, and data marketplace constructs, along with the ability to lead platform transformation initiatives and mentor engineering teams. ______________ Core Responsibilities

1. Platform Strategy & Self-Service Enablement * Define and implement a self-service data platform strategy to reduce onboarding friction.

• Lead automated provisioning of:

o Databricks workspaces (via DevHub) o Airflow/Astronomer environments o Access and entitlements (AccessCentral APIs)

• Establish isolated, stable development environments for federated teams.

• Drive platform observability by integrating metrics into tools lik e Datadog.

1. Data Discovery, Access & Governance * Architect and implement enterprise-wide data discovery and marketplace enablement.

• Drive adoption of:

o Data contracts o Metadata standards o Domain-aligned catalogs (Unity Catalog)

• Enable secure access to curated, masked datasets in dev and production environments.

• Implement tagging, access patterns, and entitlement automation.

• Partner with risk/compliance teams to enforce regulatory and governance controls (BFSI-aligned)., o Airflow DAG libraries o DBT-based transformation models

• Ensure:

o Data quality and consistency o Embedded governance and compliance policies

• Enable concurrent development using standardized patterns and environments.

1. CI/CD, Automation & Deployment * Define and enforce standard CI/CD pipelines across data products:

o Harness (or equivalent) o Databricks Asset Bundles (DAB)

• Automate:

o DAG deployments (Airflow/Astronomer) o DBT pipeline releases

• Reduce manual interventions and ensure consistent, repeatable deployments.

• Improve release reliability with feedback loops, notifications, and monitoring.

1. Data Product Publishing & Marketplace Enablement * Drive publishing of data products to:

o Unity Catalog o Enterprise Data Marketplace

• Define and enforce:

o Documentation standards o Data ownership models o Versioning and contract management

• Enable cross-domain data sharing with embedded governance and access controls.

1. Operations, Observability & Reliability * Establish a scalable operating model for data product support.

• Implement:

o Monitoring dashboards (Datadog) o Data quality frameworks o Usage and performance metrics tracking

• Improve visibility into:

o Pipeline health o Data lineage o Access and consumption patterns

• Lead incident management, root cause analysis, and escalation processes.

1. Transformation, Roadmap & Innovation * Drive execution of platform priorities such as:

o Data contract activation strategy o Domain catalog integration o Data masking in development environments o Data quality frameworks o DBT adoption and POCs

• Lead maturity uplift from:

o Manual, fragmented workflows standardized, automated paved paths

• Champion continuous improvement and innovation in developer experience.

• 10+ years of experience in Data Engineering / Data Platform roles

• Strong hands-on expertise in:

o Databricks (Delta Lake, workflows, DAG) o Apache Airflow / Astronomer o Python, SQL, DBT ,AWS

• Proven experience implementing CI/CD frameworks (Harness, GitHub Actions, Azure DevOps)

• Deep understanding of:

o Data governance (catalogs, lineage, contracts, metadata) o Data quality and masking techniques o Enterprise data platforms and marketplace ecosystems

• Experience with API-based integrations (e.g., entitlement systems like AccessCentral)

• Monitoring/observability tools (e.g., Datadog)

(part of Tata group) 3.93.9 out of 5 stars Irvine, CA $100,000 - $120,000 a year