Principal Software Engineer, Distributed Systems

We are hiring a Staff / Principal Site Reliability Engineer focused on distributed systems and platform reliability . This is not a traditional DevOps role.

You will own the architecture and engineering practices that ensure Sigma workflows execute reliably even when individual components fail.

Processes crash. Networks become unavailable. APIs time out. Workers hang. Messages may be delivered more than once. Databases experience contention. Kubernetes pods restart.

Your job is to design Sigma so those conditions are expected, detected, and automatically recovered from.

The fundamental principle is simple:

An accepted Sigma job may fail, but it should never disappear.

You will work closely with engineering leadership to evolve Sigma’s execution architecture, improve production reliability, establish observability standards, and eliminate classes of distributed-system failures before they affect customers. What You’ll Own Design and improve the systems responsible for executing long-running and scheduled infrastructure workflows.

Build patterns for:

Durable workflow execution

Idempotent operations

Automatic retries and exponential backoff

Worker leases and heartbeats

Stuck-job detection and recovery

Workflow timeouts and cancellation

Failure isolation

Dead-letter handling

Checkpointing and workflow resumption

Concurrency control

Distributed locking and lease management

Reconciliation loops

Graceful degradation during dependency failures

Evaluate and implement workflow technologies such as Temporal

or equivalent systems where appropriate.

Messaging and Event Infrastructure Own and improve Sigma’s asynchronous execution architecture.

Work extensively with technologies such as:

Kafka

Distributed consumers

Consumer groups

Message delivery semantics

Partitioning

Consumer lag

Backpressure

Retry strategies

Event ordering

Duplicate message handling

Poison-message isolation

Ensure event-processing failures cannot silently cause workflows to stop progressing.

Database Reliability Help design application patterns that remain reliable under high concurrency.

Deeply understand and troubleshoot:

PostgreSQL transactions

Row and advisory locking

Deadlocks

Lock contention

Transaction isolation

Connection pooling

Long-running transactions

Query performance

Database failover

Schema and migration safety

Partner with application engineers to identify and eliminate patterns that create production contention or reliability risks.

Kubernetes and Platform Reliability Improve the reliability of Sigma deployments running in containerized environments., Design systems that tolerate node, pod, process, and dependency failures without losing customer work.

Observability Make the state of Sigma understandable in real time.

Build and improve:

Metrics

Distributed tracing

Structured logging

Dashboards

Alerting

Workflow-level telemetry

Kafka consumer monitoring

Database performance monitoring

Worker health monitoring

Queue depth and backlog monitoring

Technologies may include:

OpenTelemetry

Grafana

Prometheus

Sentry

CloudWatch

Engineers should be able to answer questions such as:

Why is this workflow still running?

Which worker owns this task?

When did it last make progress?

Which dependency is causing the delay?

Did this operation execute once or multiple times?

Kubernetes workload architecture

Pod lifecycle and failure recovery

Resource limits and capacity planning

Autoscaling

Health checks

Graceful shutdown

Rolling deployments

High availability, Reliability Engineering Help establish engineering practices expected of mature enterprise platforms.

This includes:

Service-level objectives and indicators

Error budgets

Capacity planning

Load testing

Failure-mode testing

Chaos testing

Incident response

Root-cause analysis

Production readiness reviews, What We’re Looking For You have significant experience operating production distributed systems where reliability matters.

Strong candidates will have deep experience with several of the following:

Kubernetes

Kafka or similar event-streaming systems

PostgreSQL

Distributed systems

Asynchronous worker architectures

Workflow orchestration

Temporal, Cadence, Step Functions, Conductor, or similar systems

Python backend systems

Cloud infrastructure, At-least-once delivery

Idempotency

Leases and fencing tokens

Distributed locks

Leader election

Retry semantics

Backpressure

Eventual consistency

Transaction boundaries

Failure domains

Reconciliation

Split-brain scenarios

Distributed tracing

Most importantly, you have personally diagnosed and improved production systems experiencing real distributed-system failures.

What Success Looks Like Within your first several months, you will help Sigma establish an architecture where