Principal Software Engineer - Identity Workflows

One Identity's OneLogin team is seeking a Principal Software Engineer to lead the architecture, correctness, and evolution of identity workflow systems across our platform. These systems are responsible for executing identity and resource lifecycle operations at scale and require deep expertise in distributed systems, asynchronous processing, and reliability under real-world conditions. As a Principal Engineer, you will operate with broad technical ownership-defining standards, guiding implementation, and resolving systemic issues across provisioning workflows, mapping systems, and distributed execution pipelines. You will work across teams to ensure that identity workflows are reliable, scalable, and consistent while evolving legacy implementations toward modern, resilient architectures. This is a hands-on role that includes system design, debugging complex cross-service workflows, and driving improvements in high-throughput, asynchronous systems. You will also play a key role in mentoring engineers and raising the level of expertise in distributed systems and workflow execution across the organization. Responsibilities: Own the execution of identity and resource lifecycle workflows through scalable, reliable distributed systems that transform intent into consistent, correct outcomes at scale * Design and evolve distributed workflow systems (e.g., pipelines, queues, workers, transformation layers)

• Ensure correctness of asynchronous execution across retries, failures, and partial completion scenarios
• Define and enforce patterns for idempotency, ordering, and consistency in distributed workflows
• Optimize throughput, parallelism, and system performance across large-scale worker fleets
• Improve reliability through robust retry strategies, backpressure handling, and failure recovery mechanisms
• Define and manage ordering and consistency guarantees in distributed workflows where sequencing and dependencies matter
• Ensure observability and diagnosability of workflows, including tracing, metrics, and runtime inspection
• Ensure deterministic and correct system behavior across retries, reprocessing, and partial execution scenarios
• Identify high-impact areas for improving performance, reliability, and scalability in workflow systems
• Guide evolution of legacy batch or monolithic processing into scalable, distributed execution models

• Deep experience building and operating distributed systems with asynchronous execution models
• Strong experience with queue-based architectures (e.g., Kafka, SQS, RabbitMQ, or equivalent)
• Experience designing and operating worker-based systems at scale
• Strong understanding of eventual consistency, idempotency, and failure handling in distributed workflows
• Experience with concurrency, parallelism, and multiprocessing across distributed environments
• Experience operating systems in Kubernetes environments, including HPA-driven scaling
• Experience improving throughput, performance, and system efficiency under high load
• Experience debugging complex distributed systems involving queues, workers, and multiple services
• Experience building resilient systems with retry handling, dead-letter queues, and reprocessing strategies
• Experience evolving legacy batch or pipeline systems into modern distributed architectures
• Strong intuition for system behavior under failure, scale, and partial execution
• Experience using AI tools to analyze system behavior, debug workflows, and improve observability

Qualifications:

Software Engineering

• 8+ years of software engineering experience with ownership of distributed or high-scale production systems.
• Strong backend development experience (Ruby, Node.js, or similar).
• Solid understanding of APIs, distributed architectures, and software design principles.
• Experience designing, building, and operating distributed, asynchronous systems.

Quality Engineering

• Experience building secure, reliable systems with strong testing practices across unit, integration, and service layers.
• Strong ownership of code quality, validation, and handling complex scenarios.

Cloud & Production Systems

• Experience building and operating services in AWS or similar cloud environments.
• Good understanding of distributed systems, cloud-native architecture, and CI/CD.
• Experience with observability, production debugging, and incident response.

On-Call & Reliability

• Willingness to participate in a mandatory 24/7 on-call rotation.
• Experience responding to production incidents and contributing to reliability improvements.

AI-Assisted Development

• Experience using, or strong interest in, AI-powered development tools (e.g., GitHub Copilot, ChatGPT, Cursor).
• Ability to evaluate and refine AI-generated code, workflows, system behavior, and operational insights.

One Identity enables organizations of all sizes to better secure, manage, monitor, protect, and analyze information and infrastructure to help fuel innovation and drive their businesses forward. With team members around the globe, we intend to continue to grow revenues and add value to customers. When you join our team, you will have the opportunity to build and develop products at a scale few others can provide. Our product portfolio serves a large base of customers and we are addressing the strategic imperatives for enterprise businesses. Working with some of the most talented employees the industry has to offer, we provide enhanced career opportunities for team members to learn and grow in a rapidly changing environment. One Identity is an Equal Opportunity Employer and Prohibits Discrimination and Harassment of Any Kind: One Identity is committed to the principle of equal employment opportunity for all employees and to providing employees with a work environment free of discrimination and harassment. All employment decisions at One Identity are based on business needs, job requirements and individual qualifications, without regard to race, color, religion or belief, national, social or ethnic origin, sex (including pregnancy), age, physical, mental or sensory disability, HIV Status, sexual orientation, gender identity and/or expression, marital, civil union or domestic partnership status, past or present military service, family medical history or genetic information, family or parental status, or any other status protected by the laws or regulations in the locations where we operate. One Identity will not tolerate discrimination or harassment based on any of these characteristics. One Identity encourages applicants of all ages.