> Markdown version of [/jobs/ext/3995979-databricks-platform-engineer](https://www.wearedevelopers.com/jobs/ext/3995979-databricks-platform-engineer). Every page supports `.md` or `Accept: text/markdown`. Links point to the HTML versions so they work for humans too. Agent guide: [/agents.md](https://www.wearedevelopers.com/agents.md). --- # Databricks Platform Engineer - **Company:** Tata Consultancy Services Limited - **Location:** Irving, TX, United States - **Experience:** Experienced - **Salary:** $125,000.0 - $140,000.0 - **Contract:** Permanent contract - **Skills:** Airflow, Amazon S3, Apache HTTP Server, Audit Trail, CA Workload Automation Ae, Cloud Computing, Continuous Integration, Information Engineering, Data Governance, Software Debugging, Github, Identity and Access Management, Subnetting, Python (Programming Language), Online Transaction Processing, OpenShift, Performance Tuning, Role-Based Access Control, DataOps, SQL Databases, Data Streaming, Data Processing, Data Classification, Apache Yarn, Lightspeed, Apache Spark, Amazon Virtual Private Cloud (VPC), Build Management, Data Lakes, Kubernetes, Information Technology, Data Lineage, Collibra, AWS Glue, Apache Kafka, Bitbucket, Data Management, Machine Learning Operations, Model Registry, Cloudwatch, Amazon Simple Queue Service (SQS), Splunk, Data Pipelines, Databricks - **Published:** October 28, 2026 - **Apply:** https://jobs.localjobnetwork.com/apply/add/88726067/1 ## About the Role Must Have * Overall 10 years' experience; 4+ years of data engineering experience, with at least 2 years working with Spark in a production environment * Solid understanding of cloud networking fundamentals - VPC, subnets, VPC Endpoints, Security Groups. You do not need to be a network engineer, but you need to understand why a Spark pod's network path matters and what breaks when it is wrong * Hands-on experience with AWS services: S3, EKS, IAM, KMS, SQS, MSK * Experience with Apache Iceberg or Delta Lake in production - not just awareness, but actual table design and maintenance * Familiarity with Kubernetes - you do not need to administer a cluster, but you need to understand pod l ifecycle, resource limits, and how to read a pod describe output when something fails * Comfortable reading and writing Python and SQL at a production standard * Understanding of IAM and credential management - why static keys are unacceptable, how STS AssumeRole works, what Vault does Strong Preference * Experience with Databricks, specifically Unity Catalog, MLflow, and DLT * Experience working in a regulated financial services environment - you understand why data classification, audit trails, and access controls are not optional extras * Familiarity with Airflow (DAG design, KubernetesPodOperator, trigger chain design) * Experience with Spark performance tuning - you know what data skew looks like, when to add executors vs CPU, and how to read a Spark history server * Exposure to model governance or MLOps - understanding how MLflow Model Registry, Champion/Challenger alias management, and MRM review processes fit together Not Required But Noticed * Experience with Lakebase or other OLTP-on-lakehouse technologies * Familiarity with Collibra or similar enterprise data governance platforms * Experience designing UC group naming conventions or RBAC frameworks at scale, BACHELOR OF COMPUTER SCIENCE ## Description * Design and build batch data pipelines using Apache Spark on EKS and Databricks, processing data from NAS and upstream source systems into Iceberg tables on S3 * Own the Bronze Silver Gold DLT pipeline architecture for assigned business domains * Write and maintain DDL for Iceberg tables following the CDP naming standard (catalog.schema.table with region, LOB, and layer components) * Integrate with existing loading framework (Autosys OpenShift S3 Airflow DAG trigger chain) and extend it as the platform evolves * Work with Data Stewards and DBA team to ensure schema design meets data classification requirements (INT, PII, CPII, CSPII) * Implement and validate Unity Catalog governance controls - column masking, row-level filters, and lineage tracking - for tables under your ownership * Contribute to the migration of pipeline artifacts from DBFS to UC Volumes as part of the platform's governance uplift programmer DataOps Responsibilities * Own the operational health of pipelines in your domain - monitoring, alerting, failure triage, and restart procedures * Build and maintain job monitoring integration between Databricks / Airflow and the team's observability stack (Splunk, CloudWatch, MSK audit events) * Define and document incident runbooks for common failure patterns: EKS resource exhaustion, S3 VPC Endpoint connectivity, Spark executor out-of-memory, DLT pipeline schema evolution failures * Manage deployment of pipeline code and DAG updates through the Lightspeed CI/CD pipeline - understand what goes to S3 config bucket, what goes to the DAG bucket, and what goes to EFS * Participate in on-call rotation for production pipeline incidents * Work with the platform team on capacity planning for EKS cluster - there is no YARN queue here, resource exhaustion means job failure * Maintain data quality monitoring and reconciliation between on-prem and cloud outputs during the migration period What We Are Looking For, * Building dashboards or BI reports * Managing the Databricks platform itself - that sits with the platform team The Environment * Compute: EKS (Spark), OpenShift (on-prem loading framework), Databricks (UC, MLflow, DLT) * Storage: S3 (Iceberg), EFS (intermediate artifacts), NAS (source files) * Orchestration: Airflow (MWA), Autosys (on-prem, being phased out) * Messaging: MSK (Kafka), SQS, SNS * Catalog: HMS on Aurora migrating to AWS Glue Unity Catalog * Security: Vault (credential injection), KMS (encryption), PrivateLink (no public internet for data paths), CloudTrail (audit) * CI/CD: Lightspeed (Citi internal), Bitbucket / GitHub What Good Looks Like in This Role In the first 90 days, you should be able to: * Explain the end-to-end data flow from NAS to an Iceberg table in S3, naming every component the data touches and the network path it takes * Run a Spark job on EKS independently, debug a failure from the pod logs, and know when to look at the Spark History Server vs CloudTrail * Identify a governance gap in a UC table configuration and propose a fix In six months, you should own at least one business domain end-to-end - pipeline design, operational health, schema governance, and incident response.