Principal Engineer - Agentic AI for IT Operations
Role details
Job location
Tech stack
Job description
We are seeking a Principal Engineer to lead the design, implementation, and adoption of agentic AI solutions that transform how our Corporate Technology organization manages incidents, changes, and problems across complex, production environments including SaaS, COTS and homegrown applications which are on-prem and the cloud. This role is a key technical leader driving operational reliability, automation, and risk-aligned engineering practices across the enterprise.
You will work closely with Security, Enterprise Operations, Risk and Compliance, and application teams to architect solutions, ensure compliance with mandates, and accelerate delivery. You'll bring deep technical expertise, a collaborative mindset, and a passion for modernizing IT operations with AI-driven automation.
At this time, Ally will not sponsor a new applicant for employment authorization for this position.
The Work Itself
Agentic AI Engineering & Architecture
- Architect, develop, and evolve agentic AI solutions that automate incident management, change management, and problem management workflows.
- Build intelligent agents for detection, triage, diagnostics, communication, and remediation across production applications for SaaS, COTS and homegrown applications
- Design and implement scalable integration patterns with monitoring, logging, ticketing, automation, and observability platforms.
Operational Excellence & Production Support
- Partner with production support, SRE, and application teams to improve reliability, reduce MTTR, and automate operational runbooks.
- Establish engineering standards for safe deployment, testing, and monitoring of AI-enabled automation.
- Provide technical leadership during major incidents, ensuring agent-driven tooling and guidance supports consistent recovery.
Governance, Risk, and Compliance Leadership
- Drive completion of corporate tech risk and compliance mandates across application teams.
- Define technical controls, ensure alignment with regulatory requirements, and guide teams through implementation.
- Track program execution, identify gaps, and escalate risks with clarity and data.
Cross-Functional Partnership
- Serve as a key technical liaison across Security, Enterprise Operations, Infrastructure, Risk, and Compliance.
- Lead technical discussions, influence architectural decisions, and gain alignment across multiple stakeholder groups.
- Communicate complex concepts in a clear, approachable, and outcomes-focused way.
Leadership & Team Culture
- Mentor engineers, champion best practices, and set a high bar for engineering quality and accountability.
- Foster a collaborative, approachable, team-first culture-modeling communication, ownership, and transparency.
- Help shape long-term strategies for AI-driven IT operations across the enterprise.
Requirements
- 5+ years software engineering experience with strong depth in distributed systems, enterprise operations, or SRE.
- Bachelors Degree in Computer Science or similar field or equivalent experience.
- Hands-on experience building automation or AI-driven operational tooling (e.g., agent-based systems, LLM apps, observability-driven automation, self-healing systems).
- Expertise with incident, change, and problem management processes in large enterprise environments.
- Strong understanding of governance, risk, and compliance frameworks (technology control standards, audit readiness, policy implementation).
- Proven ability to lead cross-functional technical initiatives with diverse partners.
- Excellent communication skills; approachable, practical, and team-oriented working style.
- Ability to influence without authority and drive execution at scale.
Preferred Skills:
- Awareness the latest on AI technology
- Experience with LLM frameworks, multi-agent orchestration, or retrieval-augmented automation.
- Experience with Claude Code, IDEs with AI agents like Cursor, VSCode with Agents etc..
- Familiarity with ServiceNow, APM/log platforms (Dynatrace, Splunk) CI/CD, and cloud platforms.
- Background in FinTech or regulated industries.
Benefits & conditions
Ally's compensation program offers market-competitive base pay and pay-for-performance incentives (bonuses) based on achieving personal and company goals. But Ally's total compensation - or total rewards - extends beyond your paycheck and is designed to support and enrich your personal and professional life, including: * Time Away: competitive holiday and flexible paid-time-off, including time off for volunteering and voting. * Planning for the Future: plan for the near and long term with an industry-leading 401K retirement savings plan with matching and company contributions, student loan and 529 educational assistance programs, tuition reimbursement, and other financial well-being programs. * Supporting your Health & Well-being: flexible health and insurance options including dental and vision, pre-tax Health Savings Account with employer contributions and a total well-being program that helps you and your family stay on track physically, socially, emotionally, and financially. * Building a Family: adoption, surrogacy, and fertility support as well as parental and caregiver leave, back-up child and adult/elder day care program and childcare discounts. * Work-Life Integration: other benefits including LifeMatters® Employee Assistance Program, subsidized and discounted Weight Watchers® program and other employee discount programs.