> Markdown version of [/jobs/ext/3843130-principal-software-engineer-distributed-systems](https://www.wearedevelopers.com/jobs/ext/3843130-principal-software-engineer-distributed-systems). Every page supports `.md` or `Accept: text/markdown`. Links point to the HTML versions so they work for humans too. Agent guide: [/agents.md](https://www.wearedevelopers.com/agents.md). --- # Principal Software Engineer, Distributed Systems - **Company:** Sigma Inc. - **Location:** Lincoln, NE, United States - **Contract:** Permanent contract - **Skills:** Query Performance, Application Programming Interfaces (APIs), Cloud Computing, Databases, Concurrency Controls, DevOps, Distributed Systems, Fault Tolerance, PostgreSQL, Load Testing, Node.Js, Reliability Engineering, Workflow Management Systems, Data Logging, Autoscaling, System Availability, Grafana, Database Performance, Backend, Containerization, Kubernetes, Sentry, Apache Kafka, Cloudwatch, Stream Processing, Dynatrace - **Published:** October 20, 2026 - **Apply:** https://www.juju.com/job/23_96C9974DFA1F0ED503CB7E1BC3D6CB24 ## About the Role Kubernetes workload architecture Pod lifecycle and failure recovery Resource limits and capacity planning Autoscaling Health checks Graceful shutdown Rolling deployments High availability, Reliability Engineering Help establish engineering practices expected of mature enterprise platforms. This includes: Service-level objectives and indicators Error budgets Capacity planning Load testing Failure-mode testing Chaos testing Incident response Root-cause analysis Production readiness reviews, What We're Looking For You have significant experience operating production distributed systems where reliability matters. Strong candidates will have deep experience with several of the following: Kubernetes Kafka or similar event-streaming systems PostgreSQL Distributed systems Asynchronous worker architectures Workflow orchestration Temporal, Cadence, Step Functions, Conductor, or similar systems Python backend systems Cloud infrastructure, At-least-once delivery Idempotency Leases and fencing tokens Distributed locks Leader election Retry semantics Backpressure Eventual consistency Transaction boundaries Failure domains Reconciliation Split-brain scenarios Distributed tracing Most importantly, you have personally diagnosed and improved production systems experiencing real distributed-system failures. What Success Looks Like Within your first several months, you will help Sigma establish an architecture where ## Description We are hiring a Staff / Principal Site Reliability Engineer focused on distributed systems and platform reliability . This is not a traditional DevOps role. You will own the architecture and engineering practices that ensure Sigma workflows execute reliably even when individual components fail. Processes crash. Networks become unavailable. APIs time out. Workers hang. Messages may be delivered more than once. Databases experience contention. Kubernetes pods restart. Your job is to design Sigma so those conditions are expected, detected, and automatically recovered from. The fundamental principle is simple: An accepted Sigma job may fail, but it should never disappear. You will work closely with engineering leadership to evolve Sigma's execution architecture, improve production reliability, establish observability standards, and eliminate classes of distributed-system failures before they affect customers. What You'll Own Design and improve the systems responsible for executing long-running and scheduled infrastructure workflows. Build patterns for: Durable workflow execution Idempotent operations Automatic retries and exponential backoff Worker leases and heartbeats Stuck-job detection and recovery Workflow timeouts and cancellation Failure isolation Dead-letter handling Checkpointing and workflow resumption Concurrency control Distributed locking and lease management Reconciliation loops Graceful degradation during dependency failures Evaluate and implement workflow technologies such as Temporal or equivalent systems where appropriate. Messaging and Event Infrastructure Own and improve Sigma's asynchronous execution architecture. Work extensively with technologies such as: Kafka Distributed consumers Consumer groups Message delivery semantics Partitioning Consumer lag Backpressure Retry strategies Event ordering Duplicate message handling Poison-message isolation Ensure event-processing failures cannot silently cause workflows to stop progressing. Database Reliability Help design application patterns that remain reliable under high concurrency. Deeply understand and troubleshoot: PostgreSQL transactions Row and advisory locking Deadlocks Lock contention Transaction isolation Connection pooling Long-running transactions Query performance Database failover Schema and migration safety Partner with application engineers to identify and eliminate patterns that create production contention or reliability risks. Kubernetes and Platform Reliability Improve the reliability of Sigma deployments running in containerized environments., Design systems that tolerate node, pod, process, and dependency failures without losing customer work. Observability Make the state of Sigma understandable in real time. Build and improve: Metrics Distributed tracing Structured logging Dashboards Alerting Workflow-level telemetry Kafka consumer monitoring Database performance monitoring Worker health monitoring Queue depth and backlog monitoring Technologies may include: OpenTelemetry Grafana Prometheus Sentry CloudWatch Engineers should be able to answer questions such as: Why is this workflow still running? Which worker owns this task? When did it last make progress? Which dependency is causing the delay? Did this operation execute once or multiple times?