> Markdown version of [/jobs/ext/4581542-principal-software-engineer-guest-experience-site-reliability](https://www.wearedevelopers.com/jobs/ext/4581542-principal-software-engineer-guest-experience-site-reliability). Every page supports `.md` or `Accept: text/markdown`. Links point to the HTML versions so they work for humans too. Agent guide: [/agents.md](https://www.wearedevelopers.com/agents.md). --- # Principal Software Engineer - Guest Experience Site Reliability - **Company:** Choice Hotels International, Inc. - **Location:** Scottsdale, AZ, United States (Remote available) - **Experience:** Expert - **Contract:** Franchise - **Skills:** Java (Programming Language), JavaScript (Programming Language), Artificial Intelligence, Amazon Web Services, Apple IOS, Code Review, Relational Databases, Distributed Systems, Domain Name System (DNS), Fault Tolerance, Github, Mobile Application Software, Log Analysis, OAuth, Open Web Application Security, Reliability Engineering, Cloud Services, Akamai, Software Engineering, TypeScript, Software Vulnerability Management, Openapi, Data Logging, ReactJS, Grafana, Spring-boot, Software Security, Reliability of Systems, Git, Kotlin, Containerization, Information Technology, Deployment Automation, Performance Monitor, Graphql, Api Design, Cloudwatch, Restful APIs, Appdynamics, Software Version Control, OpenSearch, Jenkins - **Published:** November 5, 2026 - **Apply:** https://choicehotels.wd5.myworkdayjobs.com/External/job/Scottsdale-AZ---Technology--Digital-Commerce-Center/Principal-Software-Engineer---Guest-Experience-Site-Reliability_R22663 ## About the Role * Bachelor's degree in Computer Science or a related technical field, or equivalent practical experience building and operating production systems. Master's degree or equivalent experience preferred. * 10+ years of relevant experience in software development, including experience with distributed systems, cloud platforms, and API-based architecture, and experience serving as a technical lead or in an architecture role. * Strong hands-on experience with AWS public cloud services and an understanding of how managed services influence reliability, scalability, and operational behavior. * Hands-on experience building and maintaining CI/CD pipelines in Harness (Jenkins, GitHub Actions, or similar a plus), with Git-based source control and sound branching and collaboration practices. * Strong Java experience, including building and operating Java-based services (Spring Boot preferred) in cloud and containerized environments. * Proficiency building front-end applications with React and modern JavaScript or TypeScript. * Experience designing, consuming, and troubleshooting GraphQL APIs, plus familiarity with RESTful APIs and OpenAPI specifications. * Proven experience diagnosing complex production issues using observability platforms and centralized logging. Hands-on Grafana experience, including log exploration and dashboard creation, is required; experience with tools such as AppDynamics, OpenSearch, Amazon CloudWatch, or similar is a plus. * Demonstrated experience using AI-assisted tools (such as code assistants, log and trace analysis, or incident analysis tools) to improve engineering effectiveness or system reliability. * Solid understanding of Site Reliability Engineering principles, including SLOs, error budgets, incident management, and blameless postmortems. * Experience participating in an on-call rotation and leading critical incident response for high-traffic, consumer-facing systems. * Working knowledge of relational and non-relational data stores and how persistence, availability, and failure characteristics affect system resiliency. * Strong knowledge of application security best practices and awareness of relevant compliance requirements (such as PCI, OAuth, and OWASP). * Excellent written and verbal communication, with the ability to explain complex technical issues to audiences ranging from engineers to senior leaders and to facilitate productive discussion under time pressure. * Ability to deal with ambiguity, take ownership of highly complex problems from inception to completion, and lead others in the absence of formal positional power. * Preferred: Kotlin experience and experience supporting mobile applications (Android and/or iOS), including mobile release, crash, and performance monitoring. * Preferred: AWS certifications or equivalent technology certifications. * Competencies demonstrated: Technical Leadership, Problem Solving & Decision Quality, Manages Complexity, Collaboration & Communication. ## Description * Troubleshoot and resolve complex production issues across choicehotels.com, Choice's mobile apps, and call center technologies by analyzing logs, metrics, traces, and runtime signals, with Grafana as a primary tool. * Lead and facilitate cross-functional conversations during incidents and time-sensitive technical discussions, aligning engineering, product, and business partners on root cause, options, and next steps. * Participate in the on-call rotation for Guest platforms, serving as a senior escalation point and driving critical incidents to resolution. * Write secure, stable, maintainable code to fix reliability defects, build tooling and automation, and strengthen the resiliency of our applications. * Apply expertise in CDN, DNS, WAF, security, and vulnerability management to protect the environment, with experience using solutions such as Akamai. * Drive improvements to CI/CD pipelines and release safety in Harness, including automated deployment verification, rollback strategies, and pipeline guardrails. * Identify systemic risks and failure modes in AWS-hosted services, such as capacity limits, brittle error handling, poor failure propagation, and unsafe defaults, and translate them into well-scoped improvements. * Apply AI-assisted development, log analysis, and incident tools where they measurably improve root cause analysis, engineering productivity, and reliability outcomes. * Conduct trade-off analysis across alternative technical approaches and present clear pros, cons, and recommendations to peers and leadership. * Mentor and coach engineers through code reviews, pairing, and knowledge sharing, and influence engineering practices across teams without formal authority. * Partner with Solution Architects, Engineering Managers, and Mobile, Web, and Contact Center teams to define non-functional requirements for scalability, availability, and security. * Evaluate and introduce tools and technologies that improve observability, platform safety, and developer productivity, prioritizing simplicity, adoption, and long-term impact., This is an individual contributor role that will report to the Head of Digital & Loyalty Engineering. You will collaborate closely with Software Engineers, Solution Architects, Engineering Managers, and the web, mobile, and call center teams across Digital Experience to keep Choice's Guest platforms reliable and performing at their best.