> Markdown version of [/jobs/ext/4113326-principal-observability-platform-engineer](https://www.wearedevelopers.com/jobs/ext/4113326-principal-observability-platform-engineer). Every page supports `.md` or `Accept: text/markdown`. Links point to the HTML versions so they work for humans too. Agent guide: [/agents.md](https://www.wearedevelopers.com/agents.md).

---

# Principal Observability Platform Engineer

- **Company:** NSCALE, LLC
- **Location:** Houston, TX, United States
- **Experience:** Expert
- **Salary:** $190,000.0 - $300,000.0
- **Contract:** Permanent contract
- **Skills:** Artificial Intelligence, Computer Clusters, Python (Programming Language), Reliability Engineering, Ansible, Prometheus, High Performance Computing, Grafana, Kubernetes, Apache Kafka, Slurm, Machine Learning Operations, Hardware Infrastructure, Vertica, Terraform, Data Pipelines
- **Published:** October 8, 2026
- **Apply:** https://www.indeed.com/viewjob?jk=1a03972e83f812da

## About the Role

* 8+ years in SRE, infrastructure engineering, platform engineering, or observability-focused roles.
 * You've operated observability infrastructure at serious scale. You know what breaks at 10x and you design for it.
 * You have a strong bias toward simplicity. You've seen over-engineered observability stacks collapse under their own weight and you build accordingly.
 * Deep hands-on experience with a significant subset of: Prometheus, Thanos, VictoriaMetrics, Grafana, Loki, Tempo, OpenTelemetry, ClickHouse, Elastic.
 * Strong engineering fundamentals, proficient in Python, Go, or similar; comfortable owning complex systems end to end.
 * Experience with Kubernetes at scale; familiarity with GPU infrastructure or HPC environments (Slurm) is a strong plus.
 * You can architect systems, write the code, review others' work, and explain the tradeoffs clearly, all in the same week.
 * Infrastructure-as-Code is default, not optional (Terraform, Ansible, or equivalent).
 * You influence without authority. Teams want your opinion because it makes their work better.

Preferred

 * Experience with high-volume streaming pipelines for observability data (Kafka, Vector, Fluent Bit, etc.).
 * Background in AI/ML infrastructure observability: GPU utilisation, training job visibility, inference latency.
 * Prior experience defining observability strategy at an organisation level.

## Description

As a Principal/Staff Observability Platform Engineer, you'll own the technical direction of Nscale's observability platform: the systems that give us deep visibility into GPU clusters, AI workloads, and the infrastructure running them. You treat observability as a product and a discipline, not a tooling exercise. You'll set the architectural roadmap, raise the engineering bar across teams, and ensure our platform scales ahead of the business, not behind it.

You understand that complexity is a cost. Solutions that require constant babysitting don't scale, and neither does operational burden. The platforms you build should be simple to operate, easy to understand, and self-evidently correct when something goes wrong.

This isn't a "maintain and operate" role. It's a "define, build, and lead" role.

What You'll Do

 * Own the technical strategy and architecture for observability across metrics, logs, traces, and alerting at scale.
 * Drive platform decisions that have multi-year impact: tooling, data models, ingestion patterns, retention, cardinality management.
 * Identify systemic gaps before they become incidents; design platforms that make failure visible and fast to diagnose.
 * Partner with SRE, infrastructure, and AI/ML teams to embed observability natively into how Nscale builds and operates.
 * Define standards and patterns that other engineers adopt, not by mandate, but because they're clearly better.
 * Mentor and technically grow the observability team; raise the ceiling on what the team can build and own.
 * Lead incident postmortems and use them to drive durable platform improvements.
 * Evaluate and introduce tooling that meaningfully improves signal quality, operational efficiency, or scalability, and retire what doesn't.