Principal AI Platform Engineer
Role details
Job location
Tech stack
Job description
We are seeking a Principal AI Platform Engineer, to join our community. As AI agents become mission-critical in production business workflows, PEMCO needs a leader who owns the operational reliability, governance, security, and cost management of the AI layer. In year one, this is a hands-on technical leadership role. You will build systems yourself while establishing the standards and governance framework. As AI operations mature, you will build and scale a team to match operational demands.
The role spans both business AI use cases (in partnership with the Data, AI & Digital teams) and technology enablement use cases including IT operations, information security, help desk automation. You will be responsible for building and maintaining the enterprise agent marketplace, establishing production-grade observability, and ensuring governance and compliance across all AI deployments.
What You'll Be Doing:
- Own the operational lifecycle of AI agents deployed across PEMCO: deployment, monitoring, scaling, incident response, and retirement.
- Build and maintain observability for the AI layer, including cost tracking, latency, error rates, model performance, token usage, and production monitoring for agent workers.
- Manage agent orchestration infrastructure, including configuration, versioning, connection management, and tool registration. Current stack includes MCP-based orchestration and Azure OpenAI services.
- Establish runbooks and incident response procedures for AI agent failures. When an agent supporting a business workflow goes down, this role owns the recovery.
- Implement prompt governance controls and role-based model access per Information Security standards: PII exposure monitoring, prompt injection detection, and access enforcement. Contribute requirements and technical capabilities to InfoSec for AI-specific policy development.
- Build the enterprise agent marketplace: deploy and manage AI agents within an enterprise UI framework, ensuring discoverability, versioning, and access controls.
- Evaluate new model releases, track capability evolution, and make recommendations on model selection. Maintain a knowledge management layer for AI operations including decision logs, model inventories, and governance documentation.
- Contribute directly to AI governance and the AI Governance Working Group. Operationalize security, governance, and compliance standards defined by Information Security and Data & AI leadership across all AI deployments.
- Optimize AI infrastructure costs through model selection, caching strategies, batching, and token budget management.
- Partner with the Data & AI team on production readiness for models and agents. Data & AI owns model development, training, and AI governance policy. This role owns the operational deployment, monitoring, fallback behavior, graceful degradation, and knowledge layer integration.
- As the function matures, define team structure, secure headcount, and build a team.
- Demonstrate behaviors consistent with PEMCO's policies, values, code of ethics, and business conduct.
- Authentically support the PEMCO Brand and constantly are on the lookout for top talent to join us to achieve our Mission to Worry Less and Live More.
- Other duties as assigned.
Requirements
- Technical degree or equivalent practical experience.
- 5+ years in a technical operations, platform engineering, or SRE leadership role is required
- 2+ years building and deploying AI agents in production environments, including AI/ML ops (model monitoring, feature store management, RAG, vector store enablement) is required
- Experience with cloud AI services (Azure OpenAI, AWS Bedrock, Google Vertex AI, or comparable) is required
- Experience building observability and monitoring for production services is required
- Experience with identity and access management for technical platforms is required
- Demonstrated understanding of AI security risks: prompt injection, data leakage, model abuse is required
- Experience leading or building a technical team is required
- Experience with agent orchestration frameworks (MCP, LangChain, CrewAI, AutoGen, or similar)
- Experience with LLM operational patterns: token management, caching, rate limiting, fallback strategies
- Experience in a regulated industry (financial services, insurance, healthcare)
- Experience with cost optimization for cloud AI consumption
- Familiarity with AI governance frameworks (NIST AI RMF, ISO 42001, or internal frameworks)
- Experience building or operating an internal AI agent marketplace or enterprise AI access layer
- Cloud platform certifications preferred (Azure, AWS, or GCP)
- AI/ML certifications preferred but not required
- Job Specific: Translates between data teams and infrastructure teams. Comfortable in both conversations.
- Job Specific: Comfortable operating in startup mode within an established organization. Delivers value incrementally while building toward a mature platform.
- Interpersonal Skills & Empathy: Builds relationships and gets results through influence rather than authority.
- Independent: Is highly self-motivated and self-directed. The ability to work with limited direction and communicate relevant information to the appropriate levels during times of uncertainty
- MS Office: Skilled proficiency in Excel, Word, Outlook
- Precision: Is detail orientated and has a strong desire for accuracy and thoroughness
- Communicator: The ability to communicate clearly and informatively, verbally and in writing, with colleagues, customers, and the community in both technical and non-technical professional language
- Leadership & Managing Others: Establishes and communicates a compelling and inspiring vision, creates winning strategies and plans, ensures team goals are aligned with company goals; develops both self and others is required.
Benefits & conditions
The pay range for this role is shown below. Compensation decisions are determined based on an individual's qualifications, job-related knowledge, skills, and experience.
- Greater Seattle area target pay range: $154,845-$189,255. The full pay range is $129,038-$215,063.
- Outside Greater Seattle area target pay range: $136,654-$167,022. The full pay range is $113,879-$189,797.
Greater Seattle Area is defined as working within approximately 100 miles of Seattle. Outside Greater Seattle is defined as working approximately 100 miles or more from Seattle., Regular part-time PEMCO employees working at least 24 hours per week and regular full-time PEMCO employees are eligible to elect coverage under medical, dental, and vision plans for themselves and their eligible family members with generous employer premium cost shares. In addition, as a benefits-eligible employee, you are:
- covered by employer-paid basic life and accidental death & dismemberment insurance policies as well as long- and short-term disability benefit coverages.
- eligible to participate in PEMCO's 401(k) plan, which includes a generous employer match (2 for 1 on the first 6% employee pre-tax and/or Roth deferral, up to federal maximums).
PEMCO provides the following paid leave programs for benefits-eligible employees in their first year of PEMCO employment:
- Vacation accrues at a minimum rate of 10 days for new hires and increases based on a schedule to a maximum annual accrual of 25 days based on tenure.
- Granted four (4) floating holidays immediately upon hire.
- Paid holidays for the eight (8) holidays observed by PEMCO throughout the calendar year.
- Granted up to ten (10) days of sick leave immediately upon hire (pro-rated based on hire date and full-time/part-time status), which is approximately 28 hours more per year than the Washington state-required accrual.
- Paid time off for bereavement, jury duty, and employee volunteering in the community.
Other miscellaneous benefit programs offered by PEMCO include:
- Flexible Spending Accounts.
- Education Assistance Program after one year of service.
- Scholarship program for children of PEMCO employees after one year of service.
- Employee Assistance Program.
- Well-being program.
- Discretionary taxable gifts and gift cards.
- And other Perks & Benefits, including discounts on computer software and hardware, cell phone plans, and rental cars.
Other compensation, depending on role, contributions, and performance, may include:
- Discretionary bonuses.
- Tiered sales commissions and/or incentives (from 5-25% of employee's monthly sales).
- Employee referral bonuses.
- Shift differential pay.