Databricks DevOps Lead
Tata Consultancy Services Limited
Lake Forest, United States of America
3 days ago
Role details
Contract type
Permanent contract Employment type
Full-time (> 32 hours) Working hours
Regular working hours Languages
English Experience level
Senior Compensation
$ 130KJob location
Lake Forest, United States of America
Tech stack
Azure
Bash
DevOps
Distributed Systems
Github
Infrastructure as a Service (IaaS)
JUnit
Python
Key Management
Log Analysis
Powershell
Role-Based Access Control
Reliability Engineering
Azure
SonarQube
Azure
Data Logging
Scripting (Bash/Python/Go/Ruby)
Postman
Azure
Spark
Containerization
Kubernetes
Bicep
Cosmos DB
Azure
Data Management
Terraform
Data Pipelines
Azure
Docker
Key Vault
Databricks
Job description
- Mandatory skills: Databricks, Spark, IAAS, Azure, Python, Cosmos DB, Azure DevOps (GitHub, CI/CD pipelines, Boards etc.), Docker & Azure Kubernetes Service, Graffana, JUnit, Postman, SonarQube, * Ability to create/maintain CI/CD pipelines (Azure DevOps/GitHub Actions) for Databricks, Spark jobs, and Azure data components.
- Develop infrastructure as code (Terraform/Bicep/ARM) for provisioning Azure resources (Databricks, ADLS Gen2, Event Hubs, Key Vault).
- Create scripts to manage release strategies for notebooks, JARs, wheels, and configuration artifacts.
- Set up monitoring, logging, and alerting Spark jobs and data pipelines using Azure Monitor/Log Analytics or similar tools
- Coordinate with offshore team in India
Requirements
- Additional Skills: 5+ years of DevOps/SRE experience supporting data platforms or distributed systems.
- Strong experience with CI/CD pipelines (Azure DevOps/GitHub Actions) for Databricks, Spark jobs, and Azure data components. Hands?on experience with infrastructure as code (Terraform/Bicep/ARM) for provisioning Azure resources (Databricks, ADLS Gen2, Event Hubs, Key Vault). Solid understanding of build and release strategies for notebooks, JARs, wheels, and configuration artifacts. Experience setting up monitoring, logging, and alerting Spark jobs and data pipelines using Azure Monitor/Log Analytics or similar tools.
- Strong scripting skills (Python, PowerShell, or Bash) for automation and tooling.
- Good to have skills: Experience with containerization and orchestration (Docker, Kubernetes, AKS) for supporting ancillary services. Knowledge of security best practices (managed identities, secrets management, RBAC) for Azure data services. Exposure to reliability engineering concepts (error budgets, SLOs, SLI dashboards) for data workloads.
Benefits & conditions
- Discretionary Annual Incentive.
- Comprehensive Medical Coverage: Medical & Health, Dental & Vision, Disability Planning & Insurance, Pet Insurance Plans.
- Family Support: Maternal & Parental Leaves.
- Insurance Options: Auto & Home Insurance, Identity Theft Protection.
- Convenience & Professional Growth: Commuter Benefits & Certification & Training Reimbursement.
- Time Off: Vacation, Time Off, Sick Leave & Holidays.
- Legal & Financial Assistance: Legal Assistance, 401K Plan, Performance Bonus, College Fund, Student Loan Refinancing.
#LI-RJ2
Salary Range-$100,000-$130,000 a year
