Principal Site Reliability Engineer

Green Dot Corporation
Los Angeles, CA, United States
2 days ago
Apply on startup.jobs
Prepare application

Role details

Contract type
Permanent contract
Employment type
Full-time (> 32 hours)
Experience level
Expert
Experience required
5 years minimum
Compensation
$129,000 - $193,400
Working hours
Regular working hours
Job source

Tech stack

Sql Server Data Tools (SSDT) Kubernetes Security .NET Framework Amazon Web Services Systems Engineering Automation of Tests Microsoft Azure Bash Shell Cloud Computing Cyber Security Databases Continuous Integration
+52 more
Data Stores Database Security Linux DevOps Disaster Recovery Failover Clustering Fault Tolerance Github Internet Information Services (IIS) Python (Programming Language) PostgreSQL Liquibase Microsoft SQL Server SQL Azure Windows Servers Octopus Deploy PCI Data Security Standards Performance Tuning Windows PowerShell Query Optimization RabbitMQ Redis Release Management Reliability Engineering Ansible Prometheus Azure DevOps Pipelines Transact-SQL Management of Software Versions Azure Service Bus Datadog Data Logging Performance Testing Istio Grafana Database Optimization Event Driven Architecture Git Flow Kubernetes Information Technology Low Latency Deployment Automation Atlassian Tools Linkerd (Service Mesh) Cosmos DB BIG-IP Access Policy Manager (APM) Database Monitoring Cloudwatch Terraform Docker Jenkins Vulnerability Analysis

Job description

As a Principal Site Reliability Engineer, you will be the senior technical leader for reliability, release management, and database operations across Green Dot’s cloud and hybrid platforms. You will set the strategy and standards for how we build, deploy, observe, and recover our services. You will own the end-to-end release process from code merge to production, and you will make sure our data tier (SQL Server, Azure SQL, Cosmos DB, and related data stores) is highly available, performant, and resilient., Site Reliability Engineering

  • Define and drive the SRE strategy, standards, and roadmap for availability, latency, scalability, and resilience across critical payment and banking services.
  • Establish and govern SLIs, SLOs, and error budgets with product and engineering stakeholders, and report on reliability KPIs regularly to leadership.
  • Architect and maintain highly available, fault-tolerant infrastructure across Azure and AWS, including multi-region and disaster recovery (DR) designs.
  • Govern and improve the observability platform (logging, metrics, tracing, alerting) with Datadog as the primary tool, supplemented by Prometheus/Grafana and native cloud monitoring (Azure Monitor, CloudWatch).
  • Lead incident management: act as Incident Commander for high-severity events, run blameless post-incident reviews (RCAs), and track corrective actions to completion.
  • Take part in and improve the on-call rotation, including runbooks, alert quality, and escalation paths.
  • Make sure messaging and event-driven systems (e.g., Azure Service Bus, RabbitMQ) process messages efficiently, and monitor them for lag, dead-lettering, and throughput.

Proactive Reliability & Resilience

  • Lead capacity planning, performance testing, and cost optimization for cloud infrastructure.
  • Reduce toil through automation and self-service tooling.
  • Run chaos engineering, resilience testing, and DR/business continuity exercises to validate RTO/RPO targets.

Release Management

  • Own the enterprise release management process: release planning, scheduling, release calendars, change windows, and go/no-go decisions.
  • Design and implement predictable, automated deployment strategies (blue/green, canary, rolling, feature flags) across Dev, QA, UAT, Staging, and Production.
  • Architect, standardize, and maintain CI/CD pipelines in Azure DevOps and GitHub Actions (Jenkins where applicable), with automated testing, security scanning, and quality gates built in.
  • Drive GitOps practices (e.g., Argo CD) so that infrastructure and application deployments are declarative, version-controlled, and auditable.
  • Lead Change Advisory Board (CAB) participation and change management, and make sure releases have the approvals, evidence, and audit trails needed for regulated environments (e.g., SOX, PCI DSS).
  • Define branching, versioning, and environment-promotion strategies, and oversee rollback and hotfix procedures.
  • Coordinate cross-team releases with Development, QA, Product, Security, and Operations, and communicate release status, risks, and outcomes to stakeholders.
  • Track and improve DORA metrics (deployment frequency, lead time for changes, change failure rate, mean time to restore).
  • Maintain release documentation, release notes, and post-release reviews in Jira and Confluence.

Database Reliability Engineering

  • Act as the reliability lead for the data tier: Microsoft SQL Server, Azure SQL Database / Managed Instance, Cosmos DB, and related stores (e.g., PostgreSQL, Redis).
  • Design and maintain high-availability and DR solutions such as Always On Availability Groups, failover clustering, geo-replication (including Cosmos DB multi-region configuration), and backup/restore strategies with tested recovery procedures.
  • Lead database change management inside CI/CD: automated, version-controlled schema migrations (e.g., SSDT/DACPAC, Flyway, Liquibase) with safe rollback paths.
  • Drive database performance tuning: query optimization, indexing strategy, execution plan analysis, blocking/deadlock resolution, Cosmos DB partitioning and RU/s capacity, and capacity forecasting.
  • Build database observability (Datadog Database Monitoring, Query Store, Extended Events), with proactive alerts for replication lag, resource pressure, and long-running queries.
  • Partner with Information Security on database security controls, including encryption at rest and in transit, least-privilege access, auditing, and patching.
  • Automate routine DBA tasks (maintenance, patching, health checks, provisioning) using PowerShell, Python, and IaC.

Leadership & Collaboration

  • Serve as a technical authority and mentor to SRE, DevOps, and release engineers, and help build a culture of continuous learning and operational excellence.
  • Influence architecture decisions across engineering teams so that new services are designed for reliability, observability, and deployability from the start.
  • Partner with Security, Compliance, and Audit teams to support control requirements and audits with evidence.
  • Evaluate emerging tools and practices, and lead adoption where they add measurable value.

Requirements

This is a hands-on, high-impact individual contributor role. It requires deep expertise in software and systems engineering, sound judgment in a regulated environment, and the ability to influence engineering teams across the organization., * Bachelor’s degree in Computer Science, Engineering, or a related field, or equivalent experience.

  • 10+ years of experience in Site Reliability Engineering, DevOps, Release Engineering, or Infrastructure Engineering, with a record of running complex, high-availability production systems.
  • 5+ years of hands-on experience leading release management or CI/CD for multiple teams or services.
  • 5+ years of hands-on database experience with Microsoft SQL Server and/or Azure SQL, and hands-on experience with Azure Cosmos DB, including HA/DR, performance tuning, and production troubleshooting.
  • Deep expertise in Azure (required) and working knowledge of AWS.
  • Production experience with Kubernetes (AKS/EKS), Docker, and Helm.
  • Strong proficiency with Infrastructure as Code and GitOps: Terraform (required), plus Ansible and Argo CD or similar.
  • Expertise with CI/CD tooling: Azure DevOps pipelines, GitHub Actions, and Git-based workflows.
  • Expert-level observability experience with Datadog or comparable platforms (logging, APM, metrics, alerting).
  • Strong scripting and automation skills in PowerShell, Python, and Bash; strong T-SQL skills.
  • Experience with messaging and event bus technologies (Azure Service Bus, RabbitMQ, or similar).
  • Experience running incident response, RCAs, and on-call programs.
  • Strong communication skills, with the ability to explain complex technical topics to engineers and executives.

Preferred Qualifications

  • Experience in financial services, banking, payments, or fintech, including working under PCI DSS, SOX, or similar regulatory controls.
  • Experience with progressive delivery or feature-flag platforms.
  • Experience with caching technologies (Redis) and PostgreSQL.
  • Familiarity with service mesh (Istio, Linkerd) and Kubernetes security best practices.
  • Experience with .NET application environments and Windows Server/IIS as well as Linux.
  • Certifications such as Azure Solutions Architect Expert, Azure DevOps Engineer Expert, Microsoft Certified: Azure Database Administrator Associate, CKA/CKAD, AWS Solutions Architect, or ITIL.

Ability to work in the U.S. without sponsorship Ability to meet the location requirement outlined above

Benefits & conditions

The targeted base salary for this position is $129,000 to $193,400 per year. The final compensation will be determined by a number of factors such as qualifications, expertise, and the candidate’s geographical location.

About the company

We’re looking for talented professionals to join us in bringing smart money management and payment solutions to everyone’s fingertips.

This position is classified as structured hybrid, with an expectation of three (3) days per week working in the office and flexibility to work remotely on the remaining days. On-site expectations may evolve over time to support business needs, with clear communication provided in advance., We’re looking for talented professionals, anywhere in the United States, to join us in bringing smart money management and payment solutions to everyone’s fingertips. Green Dot is a financial technology and registered bank holding company. Our platforms support millions of customers and enterprise partners, and they must run securely, reliably, and in compliance with banking and payment-industry standards.

Apply for this position

This job is hosted externally. Click below to view the full posting and apply.

Apply on startup.jobs
Prepare application

Good distractions

Loading talks and stories from around this role…