Senior Data Platform / Data Product Engineering Lead

Tata Consultancy Services Limited
Irvine, United States of America
7 days ago

Role details

Contract type
Permanent contract
Employment type
Full-time (> 32 hours)
Working hours
Regular working hours
Languages
English
Experience level
Senior
Compensation
$ 120K

Job location

Irvine, United States of America

Tech stack

API
Airflow
Amazon Web Services (AWS)
Azure
Software Documentation
Continuous Integration
Directed Acyclic Graph (Directed Graphs)
Data Discovery
Information Engineering
Data Governance
Data Infrastructure
Data Masking
Data Sharing
Github
Python
Metadata Standards
SQL Databases
Systems Integration
Management of Software Versions
Enterprise Data Management
Datadog
Grafana
Data Lake
Data Lineage
Api Design
Marketplace
Databricks

Job description

We are looking for a Senior Data Platform / Data Product Engineering Lead to drive enterprise-scale data product lifecycle enablement across modern data platforms. This role will lead the design, standardization, and adoption of a paved path for data creators, enabling self-service, governed, and scalable data product development. The role requires deep expertise in Databricks, Airflow (Astronomer), CI/CD automation, data governance, and data marketplace constructs, along with the ability to lead platform transformation initiatives and mentor engineering teams., We are looking for a Senior Data Platform / Data Product Engineering Lead to drive enterprise-scale data product lifecycle enablement across modern data platforms. This role will lead the design, standardization, and adoption of a paved path for data creators, enabling self-service, governed, and scalable data product development. The role requires deep expertise in Databricks, Airflow (Astronomer), CI/CD automation, data governance, and data marketplace constructs, along with the ability to lead platform transformation initiatives and mentor engineering teams.

Core Responsibilities

  1. Platform Strategy & Self-Service Enablement

  • Define and implement a self-service data platform strategy to reduce onboarding friction.

  • Lead automated provisioning of:

o Databricks workspaces (via DevHub) o Airflow/Astronomer environments o Access and entitlements (AccessCentral APIs)

  • Establish isolated, stable development environments for federated teams.

  • Drive platform observability by integrating metrics into tools lik e Datadog.

  1. Data Discovery, Access & Governance

  • Architect and implement enterprise-wide data discovery and marketplace enablement.

  • Drive adoption of:

o Data contracts o Metadata standards o Domain-aligned catalogs (Unity Catalog)

  • Enable secure access to curated, masked datasets in dev and production environments.

  • Implement tagging, access patterns, and entitlement automation.

  • Partner with risk/compliance teams to enforce regulatory and governance controls (BFSI-aligned)., o Airflow DAG libraries o DBT-based transformation models

  • Ensure:

o Data quality and consistency o Embedded governance and compliance policies

  • Enable concurrent development using standardized patterns and environments.
  1. CI/CD, Automation & Deployment
  • Define and enforce standard CI/CD pipelines across data products:

o Harness (or equivalent) o Databricks Asset Bundles (DAB)

  • Automate:

o DAG deployments (Airflow/Astronomer) o DBT pipeline releases

  • Reduce manual interventions and ensure consistent, repeatable deployments.

  • Improve release reliability with feedback loops, notifications, and monitoring.

  1. Data Product Publishing & Marketplace Enablement
  • Drive publishing of data products to:

o Unity Catalog o Enterprise Data Marketplace

  • Define and enforce:

o Documentation standards o Data ownership models o Versioning and contract management

  • Enable cross-domain data sharing with embedded governance and access controls.
  1. Operations, Observability & Reliability

  • Establish a scalable operating model for data product support.

  • Implement:

o Monitoring dashboards (Datadog) o Data quality frameworks o Usage and performance metrics tracking

  • Improve visibility into:

o Pipeline health o Data lineage o Access and consumption patterns

  • Lead incident management, root cause analysis, and escalation processes.
  1. Transformation, Roadmap & Innovation
  • Drive execution of platform priorities such as:

o Data contract activation strategy o Domain catalog integration o Data masking in development environments o Data quality frameworks o DBT adoption and POCs

  • Lead maturity uplift from:

o Manual, fragmented workflows standardized, automated paved paths

  • Champion continuous improvement and innovation in developer experience.

Requirements

  • 10+ years of experience in Data Engineering / Data Platform roles

  • Strong hands-on expertise in:

o Databricks (Delta Lake, workflows, DAG) o Apache Airflow / Astronomer o Python, SQL, DBT ,AWS

  • Proven experience implementing CI/CD frameworks (Harness, GitHub Actions, Azure DevOps)

  • Deep understanding of:

o Data governance (catalogs, lineage, contracts, metadata) o Data quality and masking techniques o Enterprise data platforms and marketplace ecosystems

  • Experience with API-based integrations (e.g., entitlement systems like AccessCentral)

  • Monitoring/observability tools (e.g., Datadog)

Benefits & conditions

(part of Tata group) 3.93.9 out of 5 stars Irvine, CA $100,000 - $120,000 a year

Apply for this position