DevOps Engineer Observability Platform

Own the end-to-end observability architecture, including edge collection, transport, ingestion, storage, querying, visualization, and alerting across metrics, logs, and traces. Design and operate the edge telemetry layer, including OpenTelemetry Collector agents running alongside brokers at thousands of sites, managing their configuration, buffering, rollout processes, and version lifecycle. Build and operate the centralized backend on Azure Kubernetes Service (AKS), including time-series storage, log storage, trace storage, and a Grafana-based (or equivalent) visualization and alerting stack. Evaluate, select, and justify the metrics backend (e.g., VictoriaMetrics, Grafana Mimir, or Azure Monitor Managed Service for Prometheus) based on trade-offs involving cost, scalability, retention, and operational complexity. Design the telemetry ingestion and networking architecture for fleet-scale deployments using private connectivity, ensuring the ingestion layer remains healthy under aggregated load. Treat telemetry cardinality and cost optimization as core engineering concerns by developing relabeling, sampling, retention, and rollup strategies that keep the platform performant and cost-efficient as the fleet grows. Define everything as code using Terraform for infrastructure, Helm and GitOps for application and stack delivery, and GitHub Actions-based CI/CD pipelines for safe and repeatable deployments. Build the alerting and SLO framework with particular focus on signal-to-noise ratio at fleet scale-consolidating scenarios where many identical sites degrade into actionable alerts instead of generating thousands of notifications. Implement observability for the observability platform itself, detecting silent failures where a site stops reporting and continuously verifying pipeline health end-to-end. Establish retention policies, upgrade procedures, runbooks, and on-call practices for the platform, while participating in the on-call rotation. Collaborate closely with Messaging, Networking, Security, and Application teams, mentoring engineers and raising the overall standard of reliability, automation, and operational excellence.

Technical Environment

Cloud & Infrastructure: Microsoft Azure Azure Kubernetes Service (AKS) Site-to-Site VPN ExpressRoute Azure Networking Azure Storage solutions Microsoft Entra ID and Managed Identities Azure Key Vault Observability Stack: Prometheus ecosystem and PromQL Grafana (including Azure Managed Grafana) Metrics time-series databases such as:VictoriaMetrics Grafana Mimir Thanos Azure Monitor Managed Prometheus Loki for logs Tempo (or Jaeger) for distributed tracing Telemetry Pipeline: OpenTelemetry Collector Remote Write architecture Metrics, logs, and traces collection at the edge Containers & Delivery: Kubernetes Helm GitOps (Argo CD or Flux) Terraform CI/CD pipelines Programming & Automation: Python (preferred) Bash Domain Context: Event-driven messaging systems Solace PubSub+ Kafka RabbitMQ Comparable message broker technologies

Extensive experience (typically 5+ years) in DevOps, Site Reliability Engineering (SRE), Platform Engineering, or Infrastructure Engineering, with direct responsibility for production systems. Strong hands-on experience with Microsoft Azure, including operating workloads on AKS and a solid understanding of Azure networking and identity management. Deep practical knowledge of the LGTM stack and/or the VictoriaMetrics observability ecosystem, including metrics collection, PromQL, dashboards, alerting, and at least one scalable time-series database solution such as Azure Managed Prometheus, Grafana Mimir, or VictoriaMetrics. Production experience with at least two of the three observability pillars: Metrics, Logs, and Traces, along with a clear understanding of how they work together. Experience with event-driven systems and message brokers such as Solace PubSub+, Kafka, RabbitMQ, or similar technologies, including monitoring broker health, bridges, and links. Strong Infrastructure-as-Code expertise using Terraform and a code-first, GitOps-oriented mindset. Demonstrated ability to design for scale, with strong awareness of telemetry cardinality, ingestion throughput, retention strategies, and the cost implications of observability decisions. FinOps awareness and experience managing telemetry-related costs. Experience participating in on-call rotations, incident response, and creating runbooks and post-incident reviews that prevent recurrence. Excellent written communication skills and the ability to document and explain architectural decisions to both technical and non-technical stakeholders.

Eintrittsdatum Ab sofort Beschäftigungsart Anstellung, Vollzeit, 100% Homeoffice At EDEKA IT, we are at the heart of digital transformation within the EDEKA Group. As a high-performing team, we develop IT strategies, infrastructures, and services for all retail areas of the EDEKA Group throughout Germany. We pool our expertise to develop and implement innovative solutions. This is how we are driving the retail sector of tomorrow forward-perhaps soon together with you? In your role as Senior DevOps Engineer Observability Platform (m/w/d) in @@standort@@, you can make your own personal contribution to our goals., EDEKA IT is at the heart of the digital transformation within the EDEKA Group. As the company responsible for the national IT strategy and the development and provision of IT infrastructures and services for all retail areas within the group, EDEKA IT brings together expertise and drives forward the future viability of the retail sector. We believe that a love of IT connects people and makes retail powerful now and in the future. We design IT solutions that benefit both independent EDEKA retailers and customers. Through the joint development of leading IT processes - from retail to wholesale to headquarters - we are pooling our strengths and creating a foundation for actively exploiting the technological opportunities of the coming years and sustainably strengthening the retail sector.