> Markdown version of [/jobs/ext/3075374-sr-software-engineer-distributed-systems](https://www.wearedevelopers.com/jobs/ext/3075374-sr-software-engineer-distributed-systems). Every page supports `.md` or `Accept: text/markdown`. Links point to the HTML versions so they work for humans too. Agent guide: [/agents.md](https://www.wearedevelopers.com/agents.md). --- # Sr. Software Engineer - Distributed Systems - **Company:** Workday, Inc. - **Location:** Santa Clara, CA, United States (Remote available) - **Experience:** Expert - **Salary:** $190,100.0 - $285,100.0 - **Contract:** Permanent contract - **Skills:** Java (Programming Language), Artificial Intelligence, Amazon Web Services, Cloud Computing, Data Infrastructure, Data Integrity, Linux, Distributed Systems, Python (Programming Language), Open Source Technology, OpenStack, Ansible, Prometheus, Software Engineering, Datadog, Data Logging, Data Processing, Software Modules, Data Ingestion, Istio, System Availability, Large Language Models, Containerization, Kubernetes, Infrastructure Automation Frameworks, Information Technology, Bare Metal, Cloudwatch, Terraform, Dynatrace, Workday, Docker - **Published:** September 25, 2026 - **Apply:** https://workday.wd5.myworkdayjobs.com/Workday/job/USA-CA-Santa-Clara/Sr-Software-Engineer---Distributed-Systems_JR-0110113 ## About the Role Basic Qualifications - Sr Software Engineer: * 8 years of software engineering/design experience. * 7 years of coding in Python, Go, or Java. * Experience with Linux. * BS in Computer Science or a related technical field, or equivalent experience., * Ability to design, maintain, and optimize Time-Series DBs, document stores, and log collection/management systems. * Solid understanding of distributed tracing concepts (context propagation, spans, sampling) and hands-on experience with tracing tooling. * Public cloud experience (AWS/GCP), including native observability tooling such as AWS CloudWatch and GCP Cloud Operations (Stackdriver). * Experience with containerization and infrastructure automation (Docker, Kubernetes, Ansible, Chef, Terraform). * Experience with service mesh, Prometheus, and cloud-native technologies. * Hands-on experience with LLM orchestration frameworks (e.g., LangChain, LlamaIndex, Semantic Kernel) and a working knowledge of agentic AI build patterns - how agents plan, call tools, hold state, and hand off work across multi-step chains. * A knack for spotting where AI systems quietly go wrong in production: a model call that's suddenly slower than usual, a token bill creeping up for no clear reason, or answers that subtly drift in quality over time. You know how to build the dashboards, alerts, and instrumentation that catch these early - turning "the AI feels off" into a measurable, debuggable signal. * Excellent interpersonal, technical, and communication skills. * Ability to prioritize multiple tasks in a fast-paced environment. * MS Degree a plus. ## Description The Data Platform and Observability team is hiring a Senior or Mid-Level Software Engineer. We have a hybrid schedule - you'll collaborate with Workmates in the office while having the flexibility to work up to 50% remote. * Design, build, and improve critical observability services: Monitoring, Logging, Alerting, and Tracing. * Own distributed tracing end to end - instrumenting services, propagating context across service boundaries, and using trace data to understand system behavior and diagnose issues in complex, multi-hop request paths. * Build data capture and collection services using the latest technologies across multiple infrastructure types (Kubernetes, Docker, OpenStack, bare metal, etc.). * Design and develop core software modules for real-time and batch data processing. * Build metrics ingestion pipelines, alert definitions, and automation for dashboard lifecycle management. * Instrument AI-powered and automated workflows for performance, cost, and quality, including multi-step processes and tool/service orchestration. * Explore and apply AI-driven observability techniques - anomaly detection, intelligent alerting, and automated root-cause analysis - to reduce noise and speed up incident response. * Partner with product and application teams to define SLOs/SLIs and production-readiness criteria for new services and automated workflows. * Build infrastructure components and deploy them in production. * Work across all aspects of observability with a keen eye for data quality, data integrity, and data availability. * Evaluate and implement new open-source and cloud-native tools and technologies as needed. * Participate in the on-call rotation supporting the observability platform., With Flex Work, we're combining the best of both worlds: in-person time and remote. Our approach enables our teams to deepen connections, maintain a strong community, and do their best work. We know that flexibility can take shape in many ways, so rather than a number of required days in-office each week, we simply spend at least half (50%) of our time each quarter in the office or in the field with our customers, prospects, and partners (depending on role). This means you'll have the freedom to create a flexible schedule that caters to your business, team, and personal needs, while being intentional to make the most of time spent together. Those in our remote "home office" roles also have the opportunity to come together in our offices for important moments that matter. Pursuant to applicable Fair Chance law, Workday will consider for employment qualified applicants with arrest and conviction records. Workday is an Equal Opportunity Employer including individuals with disabilities and protected veterans. Workday is committed to providing reasonable accommodations for qualified individuals during our application process, in order to perform one or more essential functions of their job, as well as regarding the use of AI tools for employment decision-making to any degree. Please see below for more details including how to request an accommodation as a qualified veteran, due to a disability or for religious reasons, or as otherwise provided under applicable law. Workday prohibits taking adverse action against any candidate or employee for reporting a possible violation of this policy, requesting one or more work accommodations, exercising a privacy right, or cooperating in an investigation in accordance with applicable law. Any employee who retaliates against a candidate or employee for doing so may be subject to disciplinary action, up to and including termination of employment, to the fullest extent allowable under applicable law. ## Related Videos - [Docker network without Docker](https://www.wearedevelopers.com/videos/1418-docker-network-without-docker) - [Alibaba Big Data and Machine Learning Technology](https://www.wearedevelopers.com/videos/37-alibaba-big-data-and-machine-learning-technology) - [Docker Compose: Rediscovered](https://www.wearedevelopers.com/videos/1978-docker-compose-rediscovered) - [Rate-limiting using eBPF and Istio: How to protect your SaaS customers from themselves](https://www.wearedevelopers.com/videos/100220-rate-limiting-using-ebpf-and-istio-how-to-protect-your-saas-customers-from-themselves) - [Docker exec without Docker](https://www.wearedevelopers.com/videos/1094-docker-exec-without-docker) - [Docker build without Docker](https://www.wearedevelopers.com/videos/100114-docker-build-without-docker) ## Related Articles - [Highest Paying Tech Companies for Developers](https://www.wearedevelopers.com/magazine/220-highest-paying-tech-companies-for-developers) - [Are Software Engineer Wages Being Pushed Down? A Report on Tech Salaries](https://www.wearedevelopers.com/magazine/417-are-software-engineer-wages-being-pushed-down-a-report-on-tech-salaries) - [How Much FAANG Companies Actually Pay Software Engineers in 2025](https://www.wearedevelopers.com/magazine/230-how-much-faang-companies-actually-pay-software-engineers-in-2025) - [How Much Does a Software Engineer Make? Realistic Software Engineering Salaries](https://www.wearedevelopers.com/magazine/425-how-much-does-a-software-engineer-make-realistic-software-engineering-salaries) - [Best Countries for Software Engineers](https://www.wearedevelopers.com/magazine/267-best-countries-for-software-engineers) - [Fully Remote Software Engineer Jobs](https://www.wearedevelopers.com/magazine/447-fully-remote-software-engineer-jobs)